LSTMS

Neural News Recommendation with Collaborative News Encoding and Structural User Encoding (2021.findings-emnlp)

Challenge:	a recent study shows that LSTMs can capture syntax-sensitive generalizations such as long-distance number agreement.
Approach:	They investigate the inner mechanics of number tracking in LSTMs at the single neuron level . they find that long-distance number information is largely managed by two "number units" importantly, the behaviour of these units is partially controlled by other units to track syntactic structure .
Outcome:	The proposed model is based on a language model with a long-distance number agreement task.

Copied to clipboard

Zhiming Mao, Xingshan Zeng, Kam-Fai Wong

How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings (D19-1)

Challenge:	Existing news recommendation models encode news title and content separately without leveraging the structural correlation of user browsing histories to reflect user interests explicitly.
Approach:	They propose a news recommendation framework consisting of collaborative news encoding and structural user encode to enhance news and user representation learning.
Outcome:	The proposed framework improves the performance of news recommendation on the MIND dataset.

Copied to clipboard

Kawin Ethayarajh

The Amazing World of Neural Language Generation (2020.emnlp-tutorials)

Challenge:	Existing word embeddings were static, requiring all senses of a polysemous word to share the same representation.
Approach:	They found that the contextualized representations of all words are not isotropic in any layer of the contextualizing model.
Outcome:	The results show that the representations of all words are not isotropic in any layer of the contextualizing model.

Copied to clipboard

Yangfeng Ji, Antoine Bosselut, Thomas Wolf, Asli Celikyilmaz

Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks (2020.tacl-1)

Challenge:	Recent years have seen a paradigm shift in neural text generation due to advances in deep contextual language modeling and transfer learning.
Approach:	They will discuss how and why NLG models succeed/fail at generating coherent text.
Outcome:	This paper will discuss how and why these models succeed/fail at generating coherent text, and provide insights on several applications.

Copied to clipboard

R. Thomas McCoy, Robert Frank, Tal Linzen

Theoretical Limitations of Self-Attention in Neural Sequence Models (2020.tacl-1)

Challenge:	Inductive biases can arise from any aspect of the model architecture, study finds . we investigate which architectural factors affect how models generalize .
Approach:	They investigate which architectural factors affect generalization behavior of neural network models . they use English question formation and English tense reinflection as test cases .
Outcome:	The findings suggest that human-like generalization requires architectural syntactic structure.

Copied to clipboard

Michael Hahn

Named Entity Recognition With Parallel Recurrent Neural Networks (P18-2)

Challenge:	Existing work suggests that the computational capabilities of self-attention to model hierarchical structures are limited.
Approach:	They investigate the computational power of self-attention to model formal languages . they show strong theoretical limitations of self attention to model periodic finite-state languages unless the number of layers or heads increases with input length.
Outcome:	The proposed models can model periodic finite-state languages, nor hierarchical structure unless the number of layers or heads increases with input length.

Copied to clipboard

Andrej Žukov-Gregorič, Yoram Bachrach, Sam Coope

Converting the Point of View of Messages Spoken to Virtual Assistants (2020.findings-emnlp)

Challenge:	Named entity recognition is an important element of natural language understanding . a shift in focus has been on designing better neural architectures for solving NER .
Approach:	They propose a new architecture for named entity recognition that uses multiple LSTM units instead of a single LStm component.
Outcome:	The proposed architecture achieves state-of-the-art on the CoNLL 2003 NER dataset .

Copied to clipboard

Gunhee Lee, Vera Zu, Sai Srujana Buddi, Dennis Liang, Purva Kulkarni, Jack FitzGerald

Dialect Text Normalization to Normative Standard Finnish (D19-55)

Challenge:	Using a voice message, virtual assistants extract the message and send it to the user’s contact, rather than properly converting it to “I love you.”
Approach:	They propose to take a voice message from one user, convert it to “I love you” and deliver it to its target user.
Outcome:	The proposed system can take a voice message from one user, convert the point of view of the message, and then deliver the result to its target user.

Copied to clipboard

Niko Partanen, Mika Hämäläinen, Khalid Alnajjar

Deep Pivot-Based Modeling for Cross-language Cross-domain Transfer with Minimal Guidance (D18-1)

Challenge:	a new method for normalizing dialect transcripts is proposed for normative Finnish . dialectal Finnish is the common way of communication for people online in finnish .
Approach:	They propose a method for normalizing dialectal Finnish into the normative standard Finnish.
Outcome:	The proposed method lowers the initial word error rate of the corpus from 52.89 to 5.73 . it can be used as one processing step with many types of spoken language materials.

Copied to clipboard

Yftah Ziser, Roi Reichart

Low-Resource NLP

Challenge:	a framework for cross-domain and cross-language transfer has hardly been explored . cross-linguistic and cross language transfer methods are used for multilingual applications .
Approach:	They propose a framework that builds on pivot-based learning, structure-aware Deep Neural Networks and bilingual word embeddings to train a model on labeled data from one language pair.
Outcome:	The proposed model outperforms existing models even when trained in the lazy setup . the proposed model can be applied to nine English-German and nine English - french domain pairs without retraining .

Multi-Head Highly Parallelized LSTM Decoder for Neural Machine Translation (2021.acl-long)

Copied to clipboard

Hongfei Xu, Qiuhui Liu, Josef van Genabith, Deyi Xiong, Meng Zhang

Incremental Processing in the Age of Non-Incremental Encoders: An Empirical Assessment of Bidirectional Models for Incremental NLU (2020.emnlp-main)

Challenge:	a self-attention network can be easily parallelized at sequence level, but LSTMs are slower to train . a recent study shows that LS models require a lot of computations to perform .
Approach:	They propose to compute LSTMs at sequence level to enable sequence-level parallelization . they use a bag-of-words representation of the preceding tokens context to approximate LStms .
Outcome:	The proposed model performs better than existing models while being faster to train . the model can be trained efficiently due to the highly parallelized self-attention network .

Copied to clipboard

Brielen Madureira, David Schlangen

Coreference Resolution in Full Text Articles with BERT and Syntax-based Mention Filtering (D19-57)

Challenge:	a number of languages are processed incrementally, but the best ones do not . we test five models on various datasets and compare their performance using three incremental evaluation metrics.
Approach:	They investigate how bidirectional LSTMs and Transformers behave under incremental interfaces . they propose to use bidirectional encoders in incremental mode while retaining non-incremental quality .
Outcome:	The proposed models perform better under incremental interfaces than the "omni-directional" BERT model, which achieves better non-incremental performance, but is impacted more by the incremental access.

Copied to clipboard

Hai-Long Trieu, Anh-Khoa Duong Nguyen, Nhung Nguyen, Makoto Miwa, Hiroya Takamura, Sophia Ananiadou

How much complexity does an RNN architecture need to learn syntax-sensitive dependencies? (2020.acl-srw)

Challenge:	Existing systems for coreference resolution are difficult because of their long coreferent chains.
Approach:	They propose to use an existing span-based neural coreference resolution system as a baseline . they filter noisy mentions based on parse trees and integrate a highly expressive language model into the system .
Outcome:	The proposed system outperforms the baseline system on the CRAFT Shared Tasks 2019 task.

Copied to clipboard

Gantavya Bhatt, Hritik Bansal, Rishubh Singh, Sumeet Agarwal

Examining the Inductive Bias of Neural Language Models with Artificial Languages (2021.acl-long)

Challenge:	Long-term memory (LSTM) networks are capable of encapsulating long-range dependencies . but simple recurrent networks (SRNs) have been less successful at capturing long-term dependencies and loci of grammatical errors in an unsupervised setting.
Approach:	They propose a new architecture that incorporates the decaying nature of neuronal activations and models the excitatory and inhibitory connections in a population of neurons.
Outcome:	The proposed architecture shows competitive performance relative to LSTMs on subject-verb agreement, sentence grammaticality, and language modeling tasks.

Copied to clipboard

Jennifer C. White, Ryan Cotterell

Quantifying the Hyperparameter Sensitivity of Neural Networks for Character-level Sequence-to-Sequence Tasks (2024.eacl-long)

Challenge:	a novel method for investigating inductive biases of language models using artificial languages is proposed . we show that modern neural architectures used for language modeling are intrinsically black boxes .
Approach:	They propose a method to investigate inductive biases of language models using artificial languages . they use languages to create parallel corpora across languages that differ only in word order .
Outcome:	The proposed method shows that language models can be used to model a wide variety of languages.

Copied to clipboard

Adam Wiemerslage, Kyle Gorman, Katharina Wense

Hierarchical Multi-label Classification of Text with Capsule Networks (P19-2)

Challenge:	Neural networks are notoriously hard to interpret and slightly mysterious to researchers and practitioners alike.
Approach:	They formalize hyperparameter sensitivity using two metrics: similarity-based sensitivity and performance-based-sensitivity.
Outcome:	The transformer is more sensitive to hyperparameters according to both metrics, but not batch size . large models, multilinguality of NLP models and tasks make hyperparametric tuning more expensive .

Copied to clipboard

Rami Aly, Steffen Remus, Chris Biemann

Strong Baselines for Simple Question Answering over Knowledge Graphs with and without Neural Networks (N18-2)

Challenge:	In hierarchical multi-label classification, samples are classified into one or multiple class labels organized in a structured label hierarchy.
Approach:	They apply and compare shallow capsule networks for hierarchical multi-label text classification and introduce a new real-world scenario dataset.
Outcome:	The proposed model outperforms neural networks and non-neural network architectures on a real-world scenario dataset.

Copied to clipboard

Salman Mohammed, Peng Shi, Jimmy Lin

Multimodality

Challenge:	Existing work on simple question answering over knowledge graphs involves increasingly complex NN architectures.
Approach:	They propose to decompose the problem into entity detection, entity linking, relation prediction, evidence combination and heuristics.
Outcome:	The proposed approach outperforms existing models and benchmarks on a simple QA task.

Sensitivity as a Complexity Measure for Sequence Classification Tasks (2021.tacl-1)

Copied to clipboard

Michael Hahn, Dan Jurafsky, Richard Futrell

From Language to Language-ish: How Brain-Like is an LSTM’s Representation of Nonsensical Language Stimuli? (2020.findings-emnlp)

Challenge:	Existing complexity metrics provide limited practical insight into complexity differences between tasks.
Approach:	They propose a theoretical framework for understanding and predicting the complexity of sequence classification tasks using a new extension of the theory of Boolean function sensitivity.
Outcome:	The proposed framework predicts the complexity of sequence classification tasks using a new method . it shows that low-sensitivity functions are easier to learn for LSTMs than lexical classifiers .

Copied to clipboard

Maryam Hashemzadeh, Greta Kaufeld, Martha White, Andrea E. Martin, Alona Fyshe

Does Higher Order LSTM Have Better Accuracy for Segmenting and Labeling Sequence Data? (C18-1)

Challenge:	LSTMs are often used to measure event related potentials, but are they able to generalize to new data in a human-like way?
Approach:	They asked whether an LSTM model represents a language sample with degraded semantic or syntactic information and whether it resembles the brain's reaction to the stimuli.
Outcome:	The results suggest that LSTMs and human brain handle nonsensical data similarly.

Copied to clipboard

Yi Zhang, Xu Sun, Shuming Ma, Yang Yang, Xuancheng Ren

Certified Robustness to Programmable Transformations in LSTMs (2021.emnlp-main)

Challenge:	Existing neural models take long distance dependencies into account when predicting the tag of the current token.
Approach:	They propose a method to capture long distance tag dependencies and use them for dependency analysis.
Outcome:	The proposed model can predict multiple tags for the current token without taking dependencies between tags into account.

Copied to clipboard

Yuhao Zhang, Aws Albarghouthi, Loris D’Antoni

Robustness in NLP

Challenge:	Existing techniques for certifying robustness of LSTMs and extensions of lsts are prone to adversarial examples.
Approach:	They propose an approach to certify robustness of LSTMs and extensions of lstms . they show that their approach can train models more robust to combinations of string transformations - a key advantage of existing certification approaches .
Outcome:	The proposed approach can show high certification accuracy of the resulting models.

Towards Automating Healthcare Question Answering in a Noisy Multilingual Low-Resource Setting (P19-1)

Copied to clipboard

Jeanne E. Daniel, Willie Brink, Ryan Eloff, Charles Copley

Responsible & Trustworthy NLP

Challenge:	a study aims to automate a multilingual digital helpdesk service available via text messaging to pregnant and breastfeeding mothers in South Africa.
Approach:	They examine a multilingual digital helpdesk service available via text messaging to pregnant and breastfeeding mothers in South Africa.
Outcome:	The proposed model can accelerate response time by several orders of magnitude.

Many Hands Make Light Work: Using Essay Traits to Automatically Score Essays (2022.naacl-main)

Copied to clipboard

Rahul Kumar, Sandeep Mathias, Sriparna Saha, Pushpak Bhattacharyya

Simpler neural networks prefer subregular languages (2023.findings-emnlp)

Challenge:	In automatic essay grading, essay traits are important for scoring the essay holistically . a single-task learning system gives the best results for scoring essays holistically and scoring essay traits.
Approach:	They propose a way to score essays using a multi-task learning approach . they compare the MTL-based BiLSTM system to a single-task Learning approach based on LSTMs and BiLStms .
Outcome:	The proposed system gives better results for scoring essay holistically and scoring essay traits.

Copied to clipboard

Charles Torres, Richard Futrell

An Evaluation of Neural Machine Translation Models on Historical Spelling Normalization (C18-1)

Challenge:	Inductive biases of neural networks are still poorly understood, says dr. johansen . subregular languages are thought to form a bound on human phonological patterns .
Approach:	They apply a relaxation of L0 regularization which induces sparsity to study inductive biases of LSTMs.
Outcome:	The proposed method is based on a relaxation of L0 regularization, which induces sparsity, and a subregular language bias in LSTMs is related to the cognitive bias observed in human phonology.

Copied to clipboard

Gongbo Tang, Fabienne Cap, Eva Pettersson, Joakim Nivre

Machine Translation

Challenge:	In this paper, we apply different NMT models to the problem of historical spelling normalization for five languages . we find that NMT model is much better than SMT in terms of character error rate .
Approach:	They propose to use NMT models to solve the problem of historical spelling normalization in five languages.
Outcome:	The proposed method improves historical spelling normalization for five languages.

Long Short-Term Memory as a Dynamically Computed Element-wise Weighted Sum (P18-2)

Copied to clipboard

Omer Levy, Kenton Lee, Nicholas FitzGerald, Luke Zettlemoyer

On the Practical Computational Power of Finite Precision RNNs for Language Recognition (P18-2)

Challenge:	LSTMs were introduced to combat vanishing gradients in simple RNNs by augmenting them with gated additive recurrent connections.
Approach:	They propose to decouple the LSTM’s gates from the embedded RNN and create a new class of RNNs where the recurrence computes an element-wise weighted sum of context-independent functions of the input.
Outcome:	The proposed model performs as well as an LSTM on a range of problems, strongly suggesting that the gates are doing much more in practice than just alleviating vanishing gradients.

Copied to clipboard

Gail Weiss, Yoav Goldberg, Eran Yahav

A Comprehensive Comparison of Neural Networks as Cognitive Models of Inflection (2022.emnlp-main)

Challenge:	Recurrent Neural Networks (RNNs) are famously known to be Turing complete, but this relies on infinite precision in the states and unbounded computation time.
Approach:	They propose to use LSTM and Elman-RNN with ReLU activation to study RNNs . they show that LS and ReLU-RNns can easily implement counting behavior .
Outcome:	The LSTM and the Elman-RNN with ReLU activation are stronger than the RNN with squashing activation and the GRU.

Copied to clipboard

Adam Wiemerslage, Shiran Dudy, Katharina Kann

LSTMs Can Learn Syntax-Sensitive Dependencies Well, But Modeling Structure Makes Them Better (P18-1)

Challenge:	Neural networks are at the center of a debate about human behavior in inflectional morphology.
Approach:	They measure correlation between human judgments and neural network probabilities for unknown word inflections.
Outcome:	The proposed model for morphological inflections correlates best with human wug ratings, but not with humans.

Copied to clipboard

Adhiguna Kuncoro, Chris Dyer, John Hale, Dani Yogatama, Stephen Clark, Phil Blunsom

Adaptation of Hierarchical Structured Models for Speech Act Recognition in Asynchronous Conversation (N19-1)

Challenge:	a recent study found that language models fail to learn long-range syntax sensitive dependencies.
Approach:	They propose to use a subject-verb agreement diagnostic to determine whether language models can learn long-range syntax sensitive dependencies.
Outcome:	The proposed model outperforms left-corner and bottom-up variants in learning non-local dependencies.

Copied to clipboard

Tasnim Mohiuddin, Thanh-Tung Nguyen, Shafiq Joty

Speech & Audio in NLP

Challenge:	asynchronous domains lack large labeled datasets to train an effective speech act recognition model.
Approach:	They propose methods to leverage abundant unlabeled conversational data and available labeled data from synchronous domains to train an effective SAR model.
Outcome:	The proposed method outperforms existing methods when trained on in-domain data only.

Improved Language Modeling by Decoding the Past (P19-1)

Copied to clipboard

Siddhartha Brahma

Targeted Syntactic Evaluation of Language Models (D18-1)

Challenge:	Existing methods to improve language modeling performance are based on regularized LSTMs with a large number of parameters and training time.
Approach:	They propose a method that decodes the last token in context using the predicted distribution of the next token.
Outcome:	The proposed method improves perplexity on the Penn Treebank dataset by 1.8 points and 2.3 points on the WikiText-2 datasets.

Copied to clipboard

Rebecca Marvin, Tal Linzen

Treat us like the sequences we are: Prepositional Paraphrasing of Noun Compounds using LSTM (C18-1)

Challenge:	Recent advances have led to an explosion of neural network-based LM architectures.
Approach:	They propose to supplement perplexity with a metric that assesses whether a language model can predict the grammatical sentence more accurately than an ungrammatically-based model.
Outcome:	The proposed model performed poorly on many of the constructions.

Copied to clipboard

Girishkumar Ponkiya, Kevin Patel, Pushpak Bhattacharyya, Girish Palshikar

Tweet Stance Detection Using an Attention based Neural Ensemble Model (N19-1)

Challenge:	Using prepositions, noun compounds are interpreted in two ways: labelling and paraphrasing.
Approach:	They propose to paraphrase noun compounds using prepositions by using parallelly aligned sequences of words.
Outcome:	The proposed approach performs well on datasets manually annotated with prepositions.

Copied to clipboard

Umme Aymun Siddiqua, Abu Nowshed Chy, Masaki Aono

Who is Killed by Police: Introducing Supervised Attention for Hierarchical LSTMs (C18-1)

Challenge:	Existing deep learning approaches to stance detection in twitter are inadequate to deal with the vanishing-gradient and overfitting problems.
Approach:	They propose a neural ensemble model that adopts strengths of two LSTM variants to learn better long-term dependencies.
Outcome:	The proposed model improves on the existing deep learning models on single and multi-target stance detection datasets.

Copied to clipboard

Minh Nguyen, Thien Huu Nguyen

An Investigation of Noise in Morphological Inflection (2023.findings-acl)

Challenge:	Existing methods to identify police killings from text have not been applied to this problem . et al., 2017: finding names of people killed by police is a critical problem despite public attention .
Approach:	They propose a method to deal with multiple appearances of police names in documents . they propose hierarchical LSTMs to model multiple sentences that contain names of interests .
Outcome:	The proposed method yields state-of-the-art performance for police killing detection . it relies on hierarchical LSTMs to model the multiple sentences that contain the person names of interests .

Copied to clipboard

Adam Wiemerslage, Changbing Yang, Garrett Nicolai, Miikka Silfverberg, Katharina Kann

Morphology

Challenge:	Neural morphological inflection systems can be used for languages with very little supervised data, but are often less likely to have clean, goldstandard data.
Approach:	They propose an error taxonomy and annotation pipeline for inflection training data and propose a character-level masked language modeling (CMLM) pretraining objective.
Outcome:	The proposed pipeline is based on error taxonomy and annotation pipelines for unsupervised morphological paradigm completion.

Coloring the Black Box: What Synesthesia Tells Us about Character Embeddings (2021.eacl-main)

Copied to clipboard

Katharina Kann, Mauro M. Monsalve-Mercado

Representation Learning

Challenge:	Neural network models are difficult to understand and are considered "black boxes".
Approach:	They use grapheme–color synesthesia to study character embeddings in English . they compare graphemes to phonemes to find the most human-like character embeds .
Outcome:	The results show that grapheme-to-phoneme conversion results in the most human-like character embeddings.

CLiMP: A Benchmark for Chinese Language Model Evaluation (2021.eacl-main)

Copied to clipboard

Beilei Xiang, Changbing Yang, Yu Li, Alex Warstadt, Katharina Kann

Pretrained Transformers Improve Out-of-Distribution Robustness (2020.acl-main)

Challenge:	Linguistically informed analyses of language models (LMs) contribute to understanding and improvement of such models.
Approach:	They introduce a corpus of Chinese linguistic minimal pairs (CLiMP) to investigate what knowledge Chinese LMs acquire.
Outcome:	The proposed corpus of Chinese linguistic minimal pairs (CLiMP) covers 9 major Chinese linguist phenomena.

Copied to clipboard

Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, Dawn Song

Robustness in NLP

Challenge:	Pretrained Transformers are more effective at detecting anomalous or OOD examples, while many previous models are frequently worse than chance.
Approach:	They construct a new robustness benchmark with real distribution shifts to measure out-of-distribution generalization for seven NLP datasets and compare them to previous models.
Outcome:	The proposed model generalizations for seven datasets show that pretrained Transformers are significantly less effective at detecting anomalous or OOD examples, while many previous models are often worse than chance.

KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoning (D19-1)

Copied to clipboard

Bill Yuchen Lin, Xinyue Chen, Jamin Chen, Xiang Ren

Reasoning

Challenge:	empowering machines with the ability to perform commonsense reasoning has been seen as the bottleneck of artificial general intelligence .
Approach:	They propose a textual inference framework that uses external commonsense knowledge graphs to answer commonsensical questions.
Outcome:	The proposed framework is based on graph convolutional networks and LSTMs with a hierarchical path-based attention mechanism.

Simplicity Bias in Transformers and their Ability to Learn Sparse Boolean Functions (2023.acl-long)

Copied to clipboard

Satwik Bhattamishra, Arkil Patel, Varun Kanade, Phil Blunsom

Investigating representations of verb bias in neural language models (2020.emnlp-main)

Challenge:	Recent studies have found that Transformers struggle to model several formal languages when compared to recurrent models.
Approach:	They conduct an extensive empirical study on Boolean functions to demonstrate that Transformers are relatively more biased towards functions of low sensitivity . they also show that Transformer's generalize near perfectly even in the presence of noisy labels whereas recurrent models overfit and achieve poor generalization accuracy.
Outcome:	The results show that Transformers generalize near perfectly even in noisy Boolean functions whereas recurrent models overfit and achieve poor generalization accuracy.

Copied to clipboard

Robert Hawkins, Takateru Yamakoshi, Thomas Griffiths, Adele Goldberg

Are Transformers a Modern Version of ELIZA? Observations on French Object Verb Agreement (2021.emnlp-main)

Challenge:	Languages typically provide more than one grammatical construction to express certain types of messages.
Approach:	They propose a large benchmark dataset containing 50K human judgments for 5K distinct sentence pairs in the English dative alternation.
Outcome:	The proposed model outperforms recurrent architectures even under comparable parameter and training settings.

Copied to clipboard

Bingzhi Li, Guillaume Wisniewski, Benoit Crabbé

How Can Self-Attention Networks Recognize Dyck-n Languages? (2020.findings-emnlp)

Challenge:	Recent studies have shown that unsupervised sentence representations of neural networks encode syntactic information by observing that neural language models are able to predict the agreement between a verb and its subject.
Approach:	They propose to take an alternative look at these results by studying whether neural networks are able to build an abstract sentence representation rather than capture surface statistical regularities.
Outcome:	The proposed model can achieve high accuracy on the long-range French object-verb agreement, indicating a possible flaw in the model's syntactic ability.

Copied to clipboard

Javid Ebrahimi, Dhruv Gelda, Wei Zhang

Dissecting Span Identification Tasks with Performance Prediction (2020.emnlp-main)

Challenge:	Recent work has explored the generalized Dyck-n (Dn) languages .
Approach:	They compare the performance of two variants of self-attention networks for Dyck-n (Dn) languages with a starting symbol.
Outcome:	The proposed model can generalize to longer sequences and deeper dependencies.

Copied to clipboard

Sean Papay, Roman Klinger, Sebastian Padó

Challenge:	Span identification tasks are a staple of applied NLP, but there is little insight on how their properties influence their difficulty.
Approach:	They propose to build a model to predict span ID performance for unseen span ID tasks that can support architecture choices.
Outcome:	The proposed model predicts span ID tasks for unseen span ID task in English, and the meta model predictable span ID performance.

Language Model Evaluation Beyond Perplexity (2021.acl-long)

Copied to clipboard

Clara Meister, Ryan Cotterell

Influence Paths for Characterizing Subject-Verb Number Agreement in LSTM Language Models (2020.acl-main)

Challenge:	a nascent literature on probing language models has focused on studying linguistic phenomena.
Approach:	They propose a framework for evaluating the fit of language models to natural language tendencies.
Outcome:	The proposed framework evaluates language models to the tendencies of natural language . it shows that the models learn only a subset of the tendancies considered .

Copied to clipboard

Kaiji Lu, Piotr Mardziel, Klas Leino, Matt Fredrikson, Anupam Datta

Tensorized Embedding Layers (2020.findings-emnlp)

Challenge:	LSTMs can capture syntactic rules in artificial languages, but it is unclear whether they are as capable in natural languages.
Approach:	They propose a causal account of structural properties as carried by paths across gates and neurons of a recurrent neural network that localizes and segments the concept into a set of gate or neuron-level paths.
Outcome:	The proposed model improves on a widely-studied multi-layer LSTM language model showing that it can learn subject-verb number agreement in English.

Copied to clipboard

Oleksii Hrinchuk, Valentin Khrulkov, Leyla Mirvakhabova, Elena Orlova, Ivan Oseledets

Controlled Evaluation of Grammatical Knowledge in Mandarin Chinese Language Models (2021.emnlp-main)

Challenge:	Using the Tensor Train decomposition, embeddings layers occupy large portion of model weights, preventing their deployment in limited resource settings.
Approach:	They propose a method for parameterizing embedding layers based on the Tensor Train decomposition, which allows compressing the model significantly at the cost of a negligible drop or even a slight gain in performance.
Outcome:	The proposed method can be plugged into any model and trained end-to-end.

Copied to clipboard

Yiwen Wang, Jennifer Hu, Roger Levy, Peng Qian

Deep Attentive Sentence Ordering Network (D18-1)

Challenge:	Prior work has shown that structural supervision helps English language models learn generalizations about syntactic phenomena such as subject-verb agreement.
Approach:	They train LSTMs, Recurrent Neural Network Grammars, Transformer language models, and Transformer-parameterized generative parsing models on Mandarin Chinese datasets.
Outcome:	The proposed models learn aspects of Mandarin Chinese grammar that assess syntactic and semantic relationships.

Copied to clipboard

Baiyun Cui, Yingming Li, Ming Chen, Zhongfei Zhang

Reasoning

Challenge:	Existing methods for sentence ordering tasks rely on linguistic knowledge and are domain specific.
Approach:	They propose a deep attentive sentence ordering network which integrates self-attention mechanism with LSTMs in the encoding of input sentences.
Outcome:	The proposed model outperforms the state-of-the-art models on Sentence Ordering and Order Discrimination tasks and is shown to be highly efficient.

Spherical Latent Spaces for Stable Variational Autoencoders (D18-1)

Copied to clipboard

Jiacheng Xu, Greg Durrett

Cross-Linguistic Syntactic Evaluation of Word Prediction Models (2020.acl-main)

Challenge:	Variational autoencoders use a multivariate Gaussian latent variable to capture latent structure in data.
Approach:	They propose a variational autoencoder which uses a latent distribution instead of Gaussian . they find that the variational posterior averts the KL collapse by a fixed hyperparameter .
Outcome:	The von Mises-Fisher distribution averts the KL collapse and gives better likelihoods than Gaussian models across a range of modeling conditions.

Copied to clipboard

Aaron Mueller, Garrett Nicolai, Panayiota Petrou-Zeniou, Natalia Talmina, Tal Linzen

Pyramidal Recurrent Unit for Language Modeling (D18-1)

Challenge:	A range of studies have concluded that neural word prediction models can distinguish grammatical from ungrammatically sentences with high accuracy.
Approach:	They propose to use CLAMS to evaluate LSTM and multilingual BERT models.
Outcome:	The proposed model can learn syntax on English, French, German, Hebrew and Russian, and LSTM language models on multilingual and multilingual models.

Copied to clipboard

Sachin Mehta, Rik Koncel-Kedziorski, Mohammad Rastegari, Hannaneh Hajishirzi

Learning Word Representations with Cross-Sentence Dependency for End-to-End Co-reference Resolution (D18-1)

Challenge:	Long short term memory units are powerful tools for language modeling, but their performance can be limited by the number of parameters.
Approach:	They propose a pyramidal recurrent unit which enables learning representations in high dimensional space with more generalization power and fewer parameters.
Outcome:	The proposed model outperforms existing models with different gating mechanisms and transformations on word-level language modeling tasks.

Copied to clipboard

Hongyin Luo, Jim Glass

How poor is the stimulus? Evaluating hierarchical generalization in neural networks trained on child-directed speech (2023.acl-long)

Challenge:	Existing word embedding models generate word representations by running long short-term memory recurrent neural networks on each sentence of an input article or conversation separately.
Approach:	They propose a word embedding model that learns cross-sentence dependency . they use linear sentence linking and attentional sentence linking to learn cross-entry dependency based on context sentences .
Outcome:	The proposed model improves end-to-end co-reference resolution by taking knowledge from context sentences and the entire document.

Copied to clipboard

Aditya Yedetore, Tal Linzen, Robert Frank, R. Thomas McCoy

Multimodality

Challenge:	LSTMs and Transformers perform well at capturing the surface statistics of child-directed speech, but both model types generalize in a way consistent with an incorrect linear rule than the correct hierarchical rule.
Approach:	They train LSTMs and Transformers on text from the CHILDES corpus and evaluate what they learn about English yes/no questions.
Outcome:	The proposed models perform well at capturing the surface statistics of child-directed speech, but generalize more consistent with an incorrect linear rule than the correct hierarchical rule.

Capturing Argument Interaction in Semantic Role Labeling with Capsule Networks (D19-1)

Copied to clipboard

Xinchi Chen, Chunchuan Lyu, Ivan Titov

Mama/Papa, Is this Text for Me? (2020.coling-main)

Challenge:	State-of-the-art SRL models do not model non-local interaction between arguments . e.g., LSTMs do not allow for efficient inference .
Approach:	They propose a new approach to model interactions between arguments using capsule networks . they analyze errors in the refinement procedure by capturing intuition in a flexible way .
Outcome:	The proposed model outperforms the baseline model on all 7 languages and achieves state-of-the-art results on 5 languages including English.

Copied to clipboard

Rashedur Rahman, Gwénolé Lecorvé, Aline Étienne, Delphine Battistelli, Nicolas Béchet, Jonathan Chevelu

Challenge:	Existing methods to predict minimal age from which text can be understood for children are unresolved in computational linguistics.
Approach:	They propose a method which predicts the minimum age from which a text can be understood by a recurrent neural network.
Outcome:	The proposed method outperforms state-of-the-art models at sentence and text levels and achieves mean absolute errors of 1.86 and 2.28.

Learning Music Helps You Read: Using Transfer to Study Linguistic Structure in Language Models (2020.emnlp-main)

Copied to clipboard

Isabel Papadimitriou, Dan Jurafsky

On the Ability and Limitations of Transformers to Recognize Formal Languages (2020.emnlp-main)

Challenge:	et al., 2018a, 2018b) show that LSTMs can transfer from non-linguistic data to natural language models with different types of abstract structure.
Approach:	They propose to use transfer learning to analyze encoding of grammatical structure in neural language models.
Outcome:	The proposed method improves test performance on natural language despite no overlap in surface form or vocabulary.

Copied to clipboard

Satwik Bhattamishra, Kabir Ahuja, Navin Goyal

What Part of the Neural Network Does This? Understanding LSTMs by Measuring and Dissecting Neurons (D19-1)

Challenge:	Existing studies on LSTMs have not revealed their ability to model syntactic properties.
Approach:	They propose to build a Transformers model for a subclass of counter languages and find that their learning mechanism strongly correlates with their construction.
Outcome:	The proposed model generalizes well on counter languages and its learned mechanism correlates with it.

Copied to clipboard

Ji Xin, Jimmy Lin, Yaoliang Yu

Task Refinement Learning for Improved Accuracy and Stability of Unsupervised Domain Adaptation (P19-1)

Challenge:	Biological neural systems consist of a huge number of neurons, and can react to the environment in complicated ways.
Approach:	They propose a metric to quantify the sensitivity of neurons to each label and conduct experiments to prove it.
Outcome:	The proposed metric is based on a set of experiments that show that dropping an arbitrary neuron significantly degrades the accuracy of the model.

Copied to clipboard

Yftah Ziser, Roi Reichart

Low-Resource NLP

Challenge:	Existing approaches to domain adaptation (DA) require labeled data that can be found in only a handful of domains.
Approach:	They propose a task-refinement learning approach to solve pivot detection problems . they propose to train PBLM models with gradually increasing information exposed about each pivot .
Outcome:	The proposed approach achieves state-of-the-art accuracy in six domain adaptation setups for sentiment classification.

Quantity doesn’t buy quality syntax with neural language models (D19-1)

Copied to clipboard

Marten van Schijndel, Aaron Mueller, Tal Linzen

COGS: A Compositional Generalization Challenge Based on Semantic Interpretation (2020.emnlp-main)

Challenge:	Recurrent neural network language models can learn to predict upcoming words with remarkably low perplexity . but in syntactically complex contexts, they often assign unexpectedly high probabilities to ungrammatical words .
Approach:	They investigate whether recurrent neural networks can learn to predict upcoming words with remarkably low perplexity.
Outcome:	The proposed models perform worse than GPT and BERT in some constructions than LSTMs in other contexts.

Copied to clipboard

Najoung Kim, Tal Linzen

Explainability & Interpretability in NLP

Challenge:	Natural language is characterized by compositionality: meaning of complex expressions is constructed from the meanings of its constituent parts.
Approach:	They propose a semantic parsing dataset based on a fragment of English to assess compositional generalization abilities.
Outcome:	The proposed model can generalize meanings in a given sentence in 96–99% of the tests, but generalization accuracy is lower and the generalization sensitivity is higher.

Mutual Exclusivity Training and Primitive Augmentation to Induce Compositionality (2022.emnlp-main)

Copied to clipboard

Yichen Jiang, Xiang Zhou, Mohit Bansal