Papers by Ryan Cotterell
A Measure-Theoretic Characterization of Tight Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Language modeling is a core task in natural language processing. |
| Approach: | They propose to characterize leakage onto the set of infinite sequences by a measure-theoretic approach. |
| Outcome: | The proposed language model families are tight, meaning they will not leak . the proposed language models are based on the 'sequence leakage' hypothesis . |
Convergence and Diversity in the Control Hierarchy (2023.acl-long)
Copied to clipboard
| Challenge: | Weir has defined a hierarchy of language classes whose second member (L2) is generated by tree-adjoining grammars (TAG), linear indexed grammars, combinatory categorial grammars and head grammars. |
| Approach: | They propose to extend Weir's mechanism of control to give a definition of controllable pushdown automata (PDAs) they propose to use a stricter notion of equivalence to allow for finer-grained comparisons than weak equvalence. |
| Outcome: | The proposed language classes are d-weakly equivalent to Weir's original two-level grammar, but not d strongly equivalent. |
Uncovering Probabilistic Implications in Typological Knowledge Bases (P19-1)
Copied to clipboard
| Challenge: | linguistic typology is concerned with mapping out the relationships between languages with structural and functional properties. |
| Approach: | They propose a computational model which identifies known and new linguistic universals and uncovers them worthy of further linguistic investigation. |
| Outcome: | The proposed model outperforms baselines and knowledge base baselines. |
A Formal Perspective on Byte-Pair Encoding (2023.findings-acl)
Copied to clipboard
| Challenge: | Byte-Pair Encoding (BPE) is a popular algorithm used for tokenizing data in NLP, but the underlying optimization problem that BPE seeks to solve has not yet been laid down. |
| Approach: | They propose an algorithm which is a 1/sigma*(1-e(-sigma))-approximation of an optimal merge sequence. |
| Outcome: | The proposed algorithm improves the runtime complexity from O(NM) to O(N log M) and the lower bound of the approximation is approx0.37. |
Language Model Quality Correlates with Psychometric Predictive Power in Multiple Languages (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have found that higher quality language models provide more powerful predictors of human reading behavior, but empirical support for the QP hypothesis is mixed. |
| Approach: | They propose to test the quality–power hypothesis by using surprisal language models to test their ability to predict eye tracking data. |
| Outcome: | The proposed model is based on a set of language models with a 'quality-power' hypothesis. |
Adversarial Concept Erasure in Kernel Space (2022.emnlp-main)
Copied to clipboard
| Challenge: | Large neural networks in NLP produce real-valued representations that encode the bit of human language that they were trained on. |
| Approach: | They propose a kernelization of the recently-proposed linear concept-removal objective and propose to remove linear subspaces from the representation space. |
| Outcome: | The proposed kernelization protects against the ability of nonlinear adversaries to recover the concept. |
An Ordinal Latent Variable Model of Conflict Intensity (2023.acl-long)
Copied to clipboard
| Challenge: | Advances in automated event extraction yield massive data sets of “who did what to whom” micro-records that enable data-driven approaches to monitoring conflict. |
| Approach: | They propose a probabilistic generative model that assumes each observed event is associated with a latent intensity class. |
| Outcome: | The proposed model obtains comparatively good held-out predictive performance on a conflictual to cooperative scale. |
Estimating the Entropy of Linguistic Distributions (2022.acl-short)
Copied to clipboard
| Challenge: | Shannon entropy is a quantity of interest for linguists studying the communicative capacity of human language. |
| Approach: | They propose to use entropy estimators to estimate linguistic effects from observed data . they propose to recommend better entropic estimators for future linguistic studies . |
| Outcome: | The proposed estimators are over-estimated due to poor entropy estimators, the authors argue . they recommend the same estimators be used in future linguistic studies. |
Exact Hard Monotonic Attention for Character-Level Transduction (P19-1)
Copied to clipboard
| Challenge: | Neural sequence-to-sequence models with soft attention outperform monotonic models . current dominant method is the neural sequenceto-Sequency model with soft focus . |
| Approach: | They develop a hard attention sequence-to-sequence model that enforces strict monotonicity and learns alignment jointly. |
| Outcome: | The proposed model achieves state-of-the-art on grapheme-to-phoneme conversion and morphological inflection generation. |
On Finding the K-best Non-projective Dependency Trees (2021.acl-long)
Copied to clipboard
| Challenge: | Existing work on finding the one-best dependency tree has not extended this to finding the K-best tree. |
| Approach: | They propose to simplify the K-best spanning tree algorithm by decoding the K best dependency trees with a root constraint. |
| Outcome: | The proposed algorithm can be used to find the K-best dependency trees without root constraint. |
Discourse-Centric Evaluation of Document-level Machine Translation with a New Densely Annotated Parallel Corpus of Novels (2023.acl-long)
Copied to clipboard
| Challenge: | Several recent papers claim to have achieved human parity at sentence-level machine translation. |
| Approach: | They propose to use a dataset with rich discourse annotations to evaluate MT performance . they find that MT outputs differ fundamentally from human translations in terms of latent discourse structures. |
| Outcome: | The proposed dataset builds upon the large-scale parallel corpus BWB . it covers 15,095 entity mentions in both languages and compares them to human translations . |
Modeling the Unigram Distribution (2021.findings-acl)
Copied to clipboard
| Challenge: | a novel model for estimating the unigram distribution is proposed for a language . the model is based on the word type and word type distributions, but does not consider contextual information. |
| Approach: | They propose a neuralization of Goldwater's (2011) model for estimating the unigram distribution in a language. |
| Outcome: | The proposed model outperforms character-level models in estimating the unigram distribution in a language. |
A Fast Algorithm for Computing Prefix Probabilities (2023.acl-short)
Copied to clipboard
| Challenge: | Probabilistic context-free grammars are an important formalism in NLP . |
| Approach: | They propose to run a probabilistic context-free grammar in O(n3|N|3 + |N|4), where n is the input length and |N is the number of non-terminals in the grammar. |
| Outcome: | The proposed algorithm runs in O(n3|N|3 + |N|4), where n is the input length and |N | is the number of non-terminals in the grammar. |
On the probability–quality paradox in language generation (2022.acl-short)
Copied to clipboard
| Challenge: | a recent study suggests that mode-seeking decoding methods produce unnatural language . a higher probability text is perceived as more human-like, but this is not the case with stochastic decoding . |
| Approach: | They propose to explain why higher-probability text is perceived as more human-like . they argue that human-type language should contain an amount of information close to entropy . |
| Outcome: | The proposed model produces lower-probability text, which is perceived as more human-like . elucidation of this hypothesis supports the hypothesis . |
On the Distribution of Deep Clausal Embeddings: A Large Cross-linguistic Study (P19-1)
Copied to clipboard
| Challenge: | Empirical evidence on the prevalence and limits of embeddings has been based on either laboratory setups or corpus data of relatively limited size. |
| Approach: | They use large, dependency-parsed corpora to capture clausal embedding through dependency graphs and assess their distribution. |
| Outcome: | The results show that there is no evidence for hard constraints on embedding depth . they also show that sentences with many embeddable clauses do not display a bias towards less deep embedded sentences. |
Hexatagging: Projective Dependency Parsing as Tagging (2023.acl-short)
Copied to clipboard
| Challenge: | Using a pretrained language model, we can train language models on increasingly large amounts of data. |
| Approach: | They propose a dependency parser that constructs dependency trees by tagging words with elements from a finite set of possible tags. |
| Outcome: | The proposed approach achieves state-of-the-art performance of 96.4 LAS and 97.4 UAS on the Penn Treebank test set. |
A Tale of a Probe and a Parser (2020.acl-main)
Copied to clipboard
| Challenge: | researchers train supervised models to extract linguistic structure from output of another model . supervised model can be used to perform tasks such as part-of-speech tags or dependency trees . |
| Approach: | They compare a structural probe to a more traditional parser with a lightweight parameterisation. |
| Outcome: | The structural probe outperforms a traditional parser on seven of nine languages . the researchers found that the model outperformed the parsers by 11.1 points . |
Unsupervised Disambiguation of Syncretism in Inflected Lexicons (N18-2)
Copied to clipboard
| Challenge: | Lexical ambiguity makes it difficult to compute useful statistics of a corpus. |
| Approach: | They propose a neural network-based model that fits a prior distribution over feature bundles to a list of unigram type counts and partitions each count among different analyses of that unigrammer. |
| Outcome: | The proposed model is based on a list of unigram type counts and partitions each count among different analyses of that unigrammer. |
Exploring the Linear Subspace Hypothesis in Gender Bias Mitigation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for gender bias mitigation for word embeddings are based on pre-trained word embeds . however, the assumption that the bias subspace is linear is untested . |
| Approach: | They propose a method to isolate gender bias in word embeddings using pre-trained word embeds. |
| Outcome: | The proposed method eliminates gender bias in word embeddings but assumes bias subspace is linear . the proposed method has some drawbacks, but it is a good one for a non-linear analysis. |
Activation Scaling for Steering and Interpreting Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | a successful intervention should flip the correct with the wrong token, while remaining sparse. |
| Approach: | They propose to use activation scaling to flip the correct with the wrong token . they use gradient-based optimization to learn and evaluate a specific kind of efficient intervention . |
| Outcome: | The proposed method performs comparable with steering vectors but is much less minimal. |
Investigating Critical Period Effects in Language Acquisition through Neural Language Models (2025.tacl-1)
Copied to clipboard
| Challenge: | Scholars of human development have long debated whether these phenomena are predetermined by innately encoded developmental changes in the maturing brain or natural consequences of increased experience. |
| Approach: | They use language models to test whether CP effects are peculiar to humans . they find that LMs do not show CP when L2 exposure is delayed . scholars have long debated whether innate maturation changes predetermine CP . |
| Outcome: | The proposed model does not show CP effects when the age of exposure of L2 is delayed. |
Syntactic Control of Language Models by Posterior Inference (2025.findings-acl)
Copied to clipboard
| Challenge: | Syntactic control of generated text is crucial for many domain-specific applications of language models, where structural constraints, such as formality, grammatical correctness, or adherence to a given template, can significantly affect usability and readability. |
| Approach: | They propose to use posterior inference to enforce target constituency structure by sampling from a proposal distribution with a syntactic tagger to ensure that each generated token aligns with the desired syntakic structure. |
| Outcome: | The proposed method increases the F1 score from 12.31 (GPT2-large) and 35.33 (Llama3-8B) to about 93 in both cases without compromising the language model’s fluency. |
Measuring the Similarity of Grammatical Gender Systems by Comparing Partitions (2020.emnlp-main)
Copied to clipboard
| Challenge: | A grammatical gender system divides a lexicon into a small number of fixed categories with fixed usage across speakers. |
| Approach: | They propose to define gender systems extensionally to reduce comparisons to cluster evaluation by comparing pairwise overlaps between gender systems. |
| Outcome: | The proposed measures are based on a phylogenetic tree over extant Indo-European languages. |
Autoregressive Structured Prediction with Language Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent years have seen a paradigm shift in NLP towards using pretrained language models for a wide range of tasks. |
| Approach: | They propose to model structures as sequences of actions in autoregressive manner with PLMs . their approach allows in-structure dependencies to be learned without any loss . |
| Outcome: | The proposed approach achieves state-of-the-art on all structured prediction tasks. |
Gender Bias in Contextualized Word Embeddings (N19-1)
Copied to clipboard
| Challenge: | Existing studies show that training word embeddings in large corpora could lead to encoding societal biases present in these human-produced data. |
| Approach: | They conduct several intrinsic analyses to quantify, analyze and mitigate gender bias exhibited in ELMo’s contextualized word vectors. |
| Outcome: | The proposed method mitigates gender bias on WinoBias probing corpus and demonstrates that it can be implemented in other systems. |
If beam search is the answer, what was the question? (2020.emnlp-main)
Copied to clipboard
| Challenge: | surprisingly, beam search results on language generation tasks are low-quality . despite its high error rate, beam searches can be used to decode models with high probability . |
| Approach: | They frame beam search as the exact solution to a different decoding objective . they propose a set of decoding objectives that explicitly enforce this property . |
| Outcome: | The proposed method enforces uniform information density in text, a property motivated by cognitive science. |
Searching for More Efficient Dynamic Programs (2021.findings-emnlp)
Copied to clipboard
| Challenge: | a combinatorial model of human language often involves dynamic programming. |
| Approach: | They propose to search for a sequence of semantics-preserving transformations to improve the initial program's running time. |
| Outcome: | The proposed algorithm can find speed-ups in the initial program, and it can be used to improve it. |
Context versus Prior Knowledge in Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Existing studies have investigated how often a model will rely on prior knowledge over conflicting contextual information in answering questions. |
| Approach: | They propose two mutual information-based metrics to measure a model’s dependency on a context and on its prior about an entity. |
| Outcome: | The proposed metrics show that language models can integrate prior knowledge and new information in a predictable way across different questions and contexts. |
Please Mind the Root: Decoding Arborescences for Dependency Parsing (2020.emnlp-main)
Copied to clipboard
| Challenge: | a dependency tree has a root constraint, but only one edge may emanate from the root node. |
| Approach: | They propose an algorithm which enforces a root constraint without compromising the original runtime. |
| Outcome: | The proposed algorithm satisfies the constraint without compromising the original runtime. |
When is a Language Process a Language Model? (2024.findings-acl)
Copied to clipboard
| Challenge: | In some pathological situations, such a stochastic process may "leak" probability mass onto the set of infinite strings. |
| Approach: | They propose to view a language model as a discrete stochastic process X t : t = = t + . |
| Outcome: | The proposed conditions of tightness are generalized to language models and the literature. |
Unique Hard Attention: A Tale of Two Sides (2025.acl-short)
Copied to clipboard
| Challenge: | Understanding the expressive power of transformers has attracted attention . many studies analyze unique hard attention transformers, where attention selects a single position that maximizes the attention scores. |
| Approach: | They propose to use unique hard attention to select a single position that maximizes attention scores . they show that models with leftmost-hard attention are equivalent to soft attention . |
| Outcome: | The proposed models with leftmost-hard attention are equivalent to soft attention, suggesting they may better approximate real-world transformers than right-attention models. |
Probing for the Usage of Grammatical Number (2022.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models can be used to perform a wide array of NLP tasks, but their encoding is still a mystery. |
| Approach: | They propose a usage-based probing setup to find an encoding that the model actually uses, and propose 'a use-based approach' they propose to use a behavioral task to remove the linguistic property, and to identify which encodes are used to transfer information from a noun to its head verb. |
| Outcome: | The proposed encodings are based on a behavioral task which cannot be solved without the linguistic property. |
Determinantal Beam Search (2021.acl-long)
Copied to clipboard
| Challenge: | a new beam search approach allows for a diverse subset selection process . standard beam search does not encode interactions between candidates . |
| Approach: | They propose a beam search reformulation that casts subset selection as the subdeterminant optimization problem. |
| Outcome: | The proposed method offers competitive performance against other diverse set generation strategies while providing a more general approach to optimizing for diversity. |
An L* Algorithm for Deterministic Weighted Regular Languages (2024.emnlp-main)
Copied to clipboard
| Challenge: | Angluin (1987) introduced the active learning scheme L , where the learner interacts with an oracle by asking it queries. |
| Approach: | They propose a weighted variant of Angluin's (1987) L* algorithm for learning finite state automatas from black-box models. |
| Outcome: | The proposed algorithm learns aminimal automaton for the target language. |
Lower Bounds on the Expressivity of Recurrent Neural Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent studies of the representational capacity of neural LMs have focused on their ability to recognize formal languages. |
| Approach: | They propose to connect recurrent neural networks (RNNs) as classifiers to finite-state automatas (FSAs) and a probabilistic FSA to characterize their representational capacity. |
| Outcome: | The proposed models can express arbitrary regular LMs with linearly bounded precision. |
Equivariant Transduction through Invariant Alignment (2022.coling-1)
Copied to clipboard
| Challenge: | Existing studies have demonstrated that NLP models possess the ability to generalize compositionally, but none have tested it. |
| Approach: | They propose to use a group-equivariant neural network to encode an inductive bias for SCAN to test for this ability. |
| Outcome: | The proposed architecture outperforms existing group-equivariant approaches on the SCAN task and shows that it can generalize compositionally. |
What Kind of Language Is Hard to Language-Model? (P19-1)
Copied to clipboard
| Challenge: | a recent study suggests that language models perform poorly across languages. |
| Approach: | They propose a model that fits a paired-sample multiplicative mixed-effects model to obtain language difficulty coefficients from at least-pairwise parallel corpora. |
| Outcome: | The proposed model is able to handle missing data and is aware of inter-sentence variation. |
Finding Concept-specific Biases in Form–Meaning Associations (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to detect cross-linguistic associations are not effective, but their effects are minor. |
| Approach: | They propose a method to measure cross-linguistic associations by controlling for the influence of language family and geographic proximity within a large concept-aligned, cross-lingual lexicon. |
| Outcome: | The proposed method shows that it is small, but it is unsurprisingly small (less than 0.5% on average). |
Algorithms for Acyclic Weighted Finite-State Automata with Failure Arcs (2022.emnlp-main)
Copied to clipboard
| Challenge: | Weighted finite-state automata (WSFAs) are common formalisms in NLP. |
| Approach: | They propose an algorithm for semiring-weighted WFSAs with av-erage out symbol fractions . |
| Outcome: | The proposed algorithms are faster than the standard methods for weighted finite-state automata. |
Revisiting the Optimality of Word Lengths (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing theories of communication cost are based on the length of an utterance, but they are not based upon the frequency of the utterant. |
| Approach: | They propose a method to minimize CCH's cost by comparing language lengths to surprisal's expectation and variance-to-mean ratios. |
| Outcome: | The proposed method does not minimize CCH’s cost, but rather a lower bound, which we term CCH-lower. |
Towards Explainability in Legal Outcome Prediction Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Current legal outcome prediction models do not explain their reasoning in the real world, but human legal actors need to understand the model’s decisions. |
| Approach: | They propose a method for identifying the precedent employed by legal outcome prediction models and a taxonomy of legal precedent to compare human judges and neural models. |
| Outcome: | The proposed model learns to predict outcomes reasonably well, but its use of precedent is unlike that of human judges. |
Do Syntactic Probes Probe Syntax? Experiments with Jabberwocky Probing (2021.naacl-main)
Copied to clipboard
| Challenge: | a study of neural language models shows that syntactic probes do not properly isolate syntax. |
| Approach: | They show that syntactic probes do not properly isolate syntax . they train two probes trained on normal data and find they perform worse . |
| Outcome: | The proposed method outperforms the baseline models on the most popular models, but their lead is reduced by 53%. |
Phonotactic Complexity and Its Trade-offs (2020.tacl-1)
Copied to clipboard
| Challenge: | Existing measures of linguistic complexity are relatively coarse-see, for example, Moran and Blasi (2014) and 2 below for reviews. |
| Approach: | They propose to measure bits per phoneme using the negative log-probability of a word in a language model and a collection of 1016 basic concept words across 106 languages. |
| Outcome: | The proposed measure allows a cross-linguistic comparison of phonotactic complexity across languages. |
How (Non-)Optimal is the Lexicon? (2021.naacl-main)
Copied to clipboard
| Challenge: | lexical meanings are mapped to wordforms by usage pressures and constraints on sequences of symbols. |
| Approach: | They propose a coding-theoretic view of the lexicon and a novel generative statistical model to quantify its compressibility under various constraints. |
| Outcome: | The proposed model shows that (compositional) morphology and graphotactics can account for most of the complexity of natural codes—as measured by code length. |
Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphology (P19-1)
Copied to clipboard
| Challenge: | Gender stereotypes are manifest in most of the world's languages and are consequently propagated or amplified by NLP systems. |
| Approach: | They propose a method for converting between masculine-inflected and feminine-infflectes sentences in morphologically rich languages to reduce gender stereotyping by a factor of 2.5 without any sacrifice to grammaticality. |
| Outcome: | The proposed approach reduces gender stereotyping by 2.5 without any sacrifice to grammaticality. |
Surprise! Uniform Information Density Isn’t the Whole Story: Predicting Surprisal Contours in Long-form Discourse (2024.emnlp-main)
Copied to clipboard
| Challenge: | Uniform Information Density (UID) hypothesis posits that speakers tend to distribute information evenly across linguistic units to achieve efficient communication. |
| Approach: | They propose a functional pressure that speakers modulate information rate based on location within a hierarchically-structured model of discourse. |
| Outcome: | The proposed hypothesis posits that speakers tend to distribute information evenly across linguistic units to achieve efficient communication. |
Morphologically Aware Word-Level Translation (2020.coling-main)
Copied to clipboard
| Challenge: | Current approaches to bilingual lexicon induction (BLI) ignore inflectional morphology . current models degrade when translating less frequent inflected forms . |
| Approach: | They propose a morphologically aware probability model that models lexeme translation and inflectional morphology in a structured way. |
| Outcome: | The proposed model improves performance across 6 language pairs and in the weakly supervised setting. |
On the Relationships Between the Grammatical Genders of Inanimate Nouns and Their Co-Occurring Adjectives and Verbs (2021.tacl-1)
Copied to clipboard
| Challenge: | In many languages, nouns possess grammatical genders. |
| Approach: | They use large-scale corpora and tools from NLP and information theory to test whether there is a relationship between grammatical genders of inanimate nouns and adjectives used to describe them. |
| Outcome: | The results show that there is a statistically significant relationship between the grammatical genders of inanimate nouns and adjectives used to describe them in all six languages. |
Locally Typical Sampling (2023.tacl-1)
Copied to clipboard
| Challenge: | a discrepancy in probabilistic language generators has puzzled the language generation community for years . |
| Approach: | They propose a method for local typical sampling to enforce a conditional entropy criterion for probabilistic models. |
| Outcome: | The proposed method can provide new insights into why high-probability texts can be dull or repetitive. |
A Distributional Perspective on Word Learning in Neural Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Language models are increasingly being studied as models of human language learners. |
| Approach: | They propose a distributional approach to word learning that captures distributional knowledge and gradient preferences for the word’s appropriateness. |
| Outcome: | The proposed signatures capture knowledge of where the target word can and cannot occur as well as gradient preferences about the word’s appropriateness. |
Information Locality as an Inductive Bias for Neural Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Inductive biases are inherent in every machine learning system, argues a new study . m-local entropy measures how well symbols disambiguate the next symbol . |
| Approach: | They propose a framework that captures local uncertainty of a language by quantifying how effectively preceding symbols disambiguate the next symbol. |
| Outcome: | The proposed framework captures the local uncertainty of a language by quantifying how effectively symbols disambiguate the next symbol. |
Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTs (2021.tacl-1)
Copied to clipboard
| Challenge: | Large-scale pretraining and task-specific fine-tuning are now the standard methodology for many tasks in computer vision and natural language processing. |
| Approach: | They propose to combine two types of vision and language BERTs to create a theoretical framework that can be unified under different theoretical frameworks. |
| Outcome: | The proposed models can be classified into single-stream or dual-stream encoders and are unified under a single theoretical framework. |
Revisiting the Uniform Information Density Hypothesis (2021.emnlp-main)
Copied to clipboard
| Challenge: | The uniform information density hypothesis posits a preference among language users for utterances structured such that information is distributed uniformly across a signal. |
| Approach: | They propose to test the hypothesis by using reading time and acceptability data to examine the effect of surprisal on language comprehension and acceptabilities. |
| Outcome: | The proposed hypothesis makes predictions about language comprehension and linguistic acceptability . |
Efficient Algorithms for Recognizing Weighted Tree-Adjoining Languages (2023.emnlp-main)
Copied to clipboard
| Challenge: | a class of tree-adjoining languages can be characterized by various two-level formalisms controlled by semiring-weighted CFGs and PDAs. |
| Approach: | They propose semiring-weighted versions of controllable CFGs and PDAs . they also introduce a WPDA normal form that is analogous to Chomsky's normal form for CFG . |
| Outcome: | The proposed algorithms are more time-efficient than the previous ones for LIG, PAA, and EPDA. |
Prefix Parsing is Just Parsing (2026.acl-short)
Copied to clipboard
| Challenge: | Existing prefix parsers are typically tied to particular parsing algorithms. |
| Approach: | They propose a prefix grammar transformation that reduces prefix parsing to ordinary parsers . they propose enabling prediction of the next token by computing the next-token weight vector . |
| Outcome: | The proposed method reduces prefix parsing to ordinary parsers without modification . the transformed grammar is only a small factor larger than the input . |
A Transformer with Stack Attention (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent research suggests that transformer-based language models fail to learn basic algorithmic patterns. |
| Approach: | They propose to augment transformer-based language models with a differentiable stack-based attention mechanism that adds a level of interpretability to the model. |
| Outcome: | The proposed model can model some, but not all, deterministic context-freelanguages. |
A Deep Generative Model of Vowel Formant Typology (N18-1)
Copied to clipboard
| Challenge: | a recent study has investigated the nature of vowel inventories, i.e., which vowels a language contains . a probabilistic approach does not rule out linguistic systems completely, but it can position phenomena on a scale from very common to very improbable. |
| Approach: | They propose a generative probability model of vowel inventory typology based on acoustic information rather than discrete symbols from the international phonetic alphabet. |
| Outcome: | The proposed model uses acoustic information rather than discrete symbols from the phonetic alphabet. |
Characterizing the Expressivity of Local Attention in Transformers (2026.acl-long)
Copied to clipboard
| Challenge: | Existing studies show that global and local attention are expressively complementary. |
| Approach: | They propose to restrict global attention to a fixed-size window of preceding tokens . they also propose to add local attention to local-only transformers to increase model quality . |
| Outcome: | The proposed model outperforms the global–local transformers on formal language recognition and natural language modeling. |
A Structured Variational Autoencoder for Contextual Morphological Inflection (P18-1)
Copied to clipboard
| Challenge: | morphological inflectors typically trained on fully supervised, type-level data, but how can we improve their performance? et al., 2016: a novel latent-variable model for semi-supervised learning of inflection generation. |
| Approach: | They propose a latent-variable model for semi-supervised learning of inflection generation . they use a wake-sleep algorithm to enable posterior inference over latent variables . |
| Outcome: | The proposed model improves on 23 languages and shows 10% accuracy improvement . the proposed model is based on the wake-sleep algorithm . |
On the Idiosyncrasies of the Mandarin Chinese Classifier System (N19-1)
Copied to clipboard
| Challenge: | idiosyncrasies of the Chinese classifier system have been studied, but little work has been done to quantify them with statistical methods. |
| Approach: | They propose an information-theoretic approach to measuring idiosyncrasies in Mandarin Chinese by calculating the mutual information between the distribution over classifiers and distributions over other linguistic quantities. |
| Outcome: | The proposed method reduces uncertainty in Mandarin Chinese classifiers by knowing semantic information about nouns that they modify. |
Pareto Probing: Trading Off Accuracy for Complexity (2020.emnlp-main)
Copied to clipboard
| Challenge: | Neural networks are a pillar of modern NLP systems, but their inner workings are poorly understood. |
| Approach: | They propose a probe metric that reflects the trade-off between probe complexity and performance: the Pareto hypervolume. |
| Outcome: | The proposed probe metric conforms to accepted rankings among contextual representations, and is more complex than other probe tasks. |
Reverse-Engineering the Reader (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have sought to determine to what extent language models can serve as useful models of human cognition by aligning them to human psychometric data. |
| Approach: | They propose a method to fine-tune a language model to implicitly optimize parameters of a linear regressor that directly predicts humans’ reading times of in-context linguistic units. |
| Outcome: | The proposed technique improves language models’ psychometric predictive power but also its perplexity on held-out test data. |
On the Role of Context in Reading Time Prediction (2024.emnlp-main)
Copied to clipboard
| Challenge: | a new perspective on how readers integrate context during reading time prediction is presented . a recent study shows that the proportion of variance in reading times explained by context is smaller when context is represented by the orthogonalized predictor. |
| Approach: | They propose a technique where they project surprisal onto the orthogonal complement of frequency. |
| Outcome: | The proposed method shows that the proportion of variance in reading times explained by context is smaller when context is represented by the orthogonalized predictor. |
A Plug-and-Play Method for Controlled Text Generation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for controlling language generation are not able to produce fluent text . current methods require additional models or fine-tuning to ensure specific words are included . |
| Approach: | They propose a plug-and-play decoding method that allows for controlled language generation . they add a shift in the probability distribution over our vocabulary towards semantically similar words . |
| Outcome: | The proposed method outperforms competing methods in human evaluations and does not impact fluency. |
The Harmonic Structure of Information Contours (2025.acl-long)
Copied to clipboard
Eleftheria Tsipidi, Samuel Kiegeland, Franz Nowak, Tianyang Xu, Ethan Wilcox, Alex Warstadt, Ryan Cotterell, Mario Giulianelli
| Challenge: | Language typically does not maintain a uniform information rate, but it fluctuates around a global average . a new study suggests periodicity may be a factor in information rate oscillations . |
| Approach: | They propose a hypothesis that language does not maintain a uniform information rate . they apply harmonic regression and introduce a new extension to detect periodicity . |
| Outcome: | The proposed method reveals that language oscillates at periodic intervals across frequencies . it also offers a framework for uncovering structural pressures at various levels of linguistic granularity. |
Can Transformers Learn n-gram Language Models? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing work has tested transformers' ability to represent formal languages, but language models are not classifiers of strings but rather distributions over them. |
| Approach: | They relate transformers' ability to learn random n-gram language models to ngram language model (LM) they find add- smoothing outperforms transformers on the former, while transformers perform better on the latter . |
| Outcome: | The proposed models outperform classical methods designed to learn n-gram LMs, while transformers perform better on the latter. |
Best-First Beam Search (2020.tacl-1)
Copied to clipboard
| Challenge: | Currently, beam search is the default for decoding structured predictors . however, little work has been done to speed up beam search itself . |
| Approach: | They propose a beam search algorithm that prunes the scoring function to a monotonic sequence length, which allows for safe pruning of hypotheses that cannot be in the final set of hypothecies. |
| Outcome: | The proposed method can be implemented up to 10x faster in practice. |
On the Intersection of Context-Free and Regular Languages (2023.eacl-main)
Copied to clipboard
| Challenge: | We generalize the Bar-Hillel intersection construction so that the given WFSA may contain -arcs. |
| Approach: | They propose a construction that generalizes the Bar- Hillel in the case the desired automaton has -arcs and generalize the weighted extension so that the given WFSA may contain arcs. |
| Outcome: | The proposed construction can encode the structure of both the input automaton and grammar while retaining the asymptotic size of the original construction. |
Exact Paired-Permutation Testing for Structured Test Statistics (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to evaluate test statistic are Monte Carlo approximations which use a summation over all 2 N possible swaps. |
| Approach: | They propose an exact algorithm for the paired-permutation test for a family of structured test statistics. |
| Outcome: | The proposed algorithm is 10x faster than the Monte Carlo approximation with 20000 samples on a common dataset. |
Meaning to Form: Measuring Systematicity as Information (P19-1)
Copied to clipboard
| Challenge: | A longstanding debate in semiotics centers on the relationship between linguistic signs and their corresponding semantics: is there an arbitrary relationship between word forms and their meaning, or does some systematic phenomenon pervade? |
| Approach: | They propose to quantify the systematicity of the sign using mutual information and recurrent neural networks to examine 106 languages. |
| Outcome: | The proposed model reduces entropy in a word form conditioned on its semantic representation and recovers English examples of systematic affixes. |
It’s All in the Name: Mitigating Gender Bias with Name-Based Counterfactual Data Substitution (D19-1)
Copied to clipboard
| Challenge: | Existing attempts to mitigate gender bias rely on operationalisation of gender bias as a projection over a linear subspace. |
| Approach: | They propose to operationalise gender bias as a linear subspace and augmented a corpus to remove bias by swapping all inherently-gendered words in the copy. |
| Outcome: | The proposed approach outperforms projection-based methods at the task of drawing non-biased gender analogies by an average of 19% across both corpora. |
A Spatio-Temporal Point Process for Fine-Grained Modeling of Reading Behavior (2025.acl-long)
Copied to clipboard
| Challenge: | Standard models that focus on fixation durations ignore spatial dynamics of reading . authors propose a model that captures how long fixations last, where they land and when . |
| Approach: | They propose a generative model that captures how long fixations last and where they land and when they occur. |
| Outcome: | The proposed model exhibits higher likelihood on held-out reading data than baselines. |
Linear-Time Modeling of Linguistic Structure: An Order-Theoretic Perspective (2023.emnlp-main)
Copied to clipboard
| Challenge: | a new framework for structured prediction is developed for natural language processing . a systematic approach to structured prediction requires exhaustive pair-wise comparisons of tokens . |
| Approach: | They propose a method that models the relationship between pairs of tokens in a string . they use a parallel method that predicts real numbers for each token in . |
| Outcome: | The proposed method doubles the speed of graph-based dependency parsers and brings 10-times speed-up over graph-driven dependency parses. |
Log-linear Guardedness and its Implications (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods for erasing human-interpretable concepts from neural representations that assume linearity are not fully understood. |
| Approach: | They define linear guardedness as the inability of an adversary to predict the concept directly from the representation . they show that a log-linear model can be constructed that indirectly recovers the concept . |
| Outcome: | The proposed model can be constructed that indirectly recovers the erased concept in some cases. |
Disambiguatory Signals are Stronger in Word-initial Positions (2021.eacl-main)
Copied to clipboard
| Challenge: | lexical studies of word processing and lexicon access provide evidence of preferred nature of word-initial versus word-final segments . conjecture that languages have evolved to provide more information earlier in words is based on existing methods . |
| Approach: | They propose to use a new method to assess the informativeness of word-initial versus word-final segments. |
| Outcome: | The proposed measures avoid the confounds found in existing methods. |
On Parsing as Tagging (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to reduce constituency parsing to tagging are based on linearization, learning, and decoding . linearization of the derivation tree is the most critical factor in achieving accurate parsers as taggers . |
| Approach: | They propose a pipeline with three steps for reducing constituency parsing to tagging . they find that linearization and learning are critical factors for accurate parsers . |
| Outcome: | The proposed pipelines are linearized, learning, and decoded, and have three steps to achieve accurate parsing as taggers. |
Naturalistic Causal Probing for Morpho-Syntax (2023.tacl-1)
Copied to clipboard
| Challenge: | Existing methods for probing are limited and lack understanding of their limitations and weaknesses. |
| Approach: | They propose a strategy for input-level intervention on naturalistic sentences . they use morpho-syntactic features of a sentence to intervene on the rest of the sentence . |
| Outcome: | The proposed approach allows for input-level intervention on naturalistic sentences while keeping the rest of the sentence unchanged. |
Probing as Quantifying Inductive Bias (2022.acl-long)
Copied to clipboard
| Challenge: | Pre-trained contextual representations have led to performance improvements on downstream tasks. |
| Approach: | They propose a Bayesian framework that quantifies the amount of inductive bias that the representations encode on a specific task. |
| Outcome: | The proposed framework alleviates many problems found in probing and can offer better inductive bias than BERT. |
Can Language Models Learn Typologically Implausible Languages? (2026.tacl-1)
Copied to clipboard
| Challenge: | Language models provide a naturalistic framework for studying artificial language learning . authors: typological universals and tendencies are thought to be caused by a learning bias . |
| Approach: | They propose to train LMs on highly naturalistic counterfactual versions of English and Japanese . they show that LM learn subtly implausible languages more slowly . |
| Outcome: | The proposed language models learn subtly implausible languages more slowly compared to human models . the findings suggest that LMs exhibit typologically aligned learning preferences . |
On Decoding Strategies for Neural Text Generators (2022.tacl-1)
Copied to clipboard
| Challenge: | a recent study suggests that decoding strategies may be more important than the model architecture itself when generating text from probabilistic models. |
| Approach: | They propose to measure changes in attributes of generated text as a function of decoding strategy and task using human and automatic evaluation. |
| Outcome: | The proposed study shows that decoding strategies do not always transfer across tasks . authors show that the differences in attributes are not always consistent across tasks, they say . |
On the Role of Negative Precedent in Legal Outcome Prediction (2023.tacl-1)
Copied to clipboard
| Challenge: | Legal outcome prediction is an increasingly popular task in AI. |
| Approach: | They propose to use the dynamics of a court process to develop two new models inspired by the dynamics. |
| Outcome: | The proposed model improves positive outcome prediction score to 77.15 F1 and doubles negative outcome prediction performance to 24.01 F1. |
A Simple Joint Model for Improved Contextual Neural Lemmatization (N19-1)
Copied to clipboard
| Challenge: | False positive: a core NLP task of lemmatization seeks to map multiple forms of English verbs to a canonical one, known as the lemma. |
| Approach: | They propose a joint neural model for lemmatization and morphological tagging that achieves state-of-the-art results on 20 languages from the Universal Dependencies corpora. |
| Outcome: | The proposed model achieves state-of-the-art results on 20 languages from the Universal Dependencies corpora. |
On the Proper Treatment of Tokenization in Psycholinguistics (2024.emnlp-main)
Copied to clipboard
| Challenge: | Language models are used in computational psycholinguistics to test theories that relate the surprisal of a region of interest to its cognitive cost experienced by readers. |
| Approach: | They propose to marginalize token-level language models into character-level ones before they are used in psycholinguistic studies. |
| Outcome: | The proposed model over token strings is better than character-level model, the authors show . the proposed model marginalizes token-level models into character-based models before they are used in psycholinguistic studies. |
A Cross-Linguistic Pressure for Uniform Information Density in Word Order (2023.tacl-1)
Copied to clipboard
Thomas Hikaru Clark, Clara Meister, Tiago Pimentel, Michael Hahn, Ryan Cotterell, Richard Futrell, Roger Levy
| Challenge: | a recent study has compared real and counterfactual word orders, but one functional pressure has been overlooked . a study of 10 typologically diverse languages shows that real word orders have greater uniformity than reverse word orders . |
| Approach: | They propose to test whether a pressure for UID may have influenced word order patterns cross-linguistically. |
| Outcome: | The proposed model shows that real orders have greater uniformity than reverse orders among SVO languages. |
UniMorph 2.0: Universal Morphology (L18-1)
Copied to clipboard
Christo Kirov, Ryan Cotterell, John Sylak-Glassman, Géraldine Walther, Ekaterina Vylomova, Patrick Xia, Manaal Faruqui, Sabrina J. Mielke, Arya McCarthy, Sandra Kübler, David Yarowsky, Jason Eisner, Mans Hulden
| Challenge: | The Universal Morphology project is a collaborative effort to improve how NLP handles complex morphology across the world's languages. |
| Approach: | They propose to use a universal tagset to annotate morphological data using a schema that includes a lemma and a bundle of morphology features. |
| Outcome: | The project releases annotated morphological data using a universal tagset, the UniMorph schema. |
On Efficiently Representing Regular Languages as RNNs (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent work by Hewitt et al. (2020) provides an interpretation of the empirical success of recurrent neural networks (RNNs) as language models (LMs). |
| Approach: | They generalize their construction and show that RNNs can efficiently represent a larger class of LMs than previously claimed. |
| Outcome: | The results suggest that RNNs can represent a larger class of LMs than previously claimed . |
On the Machine Learning of Ethical Judgments from Natural Language (2022.naacl-main)
Copied to clipboard
| Challenge: | a recent study examines the morality of NLP models that can take in arbitrary text and output a moral judgment . a Delphi project is a popular system for moral prediction, but it has received criticism . |
| Approach: | They propose to critique NLP methods for automating ethical decision-making . they examine a nascent task of predicting moral and ethical decisions from text . |
| Outcome: | The proposed model is unsafe at any accuracy, the authors argue . they argue that the proposed model could be useful in NLP, but not in AI. |
Efficient Sampling of Dependency Structure (2021.emnlp-main)
Copied to clipboard
| Challenge: | Spanning trees are a fundamental model of dependency structure in natural language processing, syntactic dependency trees. |
| Approach: | They propose to use a spanning tree sampling algorithm to faithfully sample dependency trees from a graph subject to a root constraint. |
| Outcome: | The proposed sampling algorithm can sample K trees without replacement in O(K N3 + K2 N) time. |
A Corpus for Large-Scale Phonetic Typology (2020.acl-main)
Copied to clipboard
Elizabeth Salesky, Eleanor Chodroff, Tiago Pimentel, Matthew Wiesner, Ryan Cotterell, Alan W Black, Jason Eisner
| Challenge: | Existing multilingual speech corpora have limited data in many languages . existing corpus is limited to a small number of languages with available data . |
| Approach: | They propose a large-scale phonetic typology corpus with phoneme-level labels and phoneme alignments in 690 readings spanning 635 languages. |
| Outcome: | The proposed corpus covers 635 languages and includes acoustic-phonetic measures of vowels and sibilants. |
Are All Languages Equally Hard to Language-Model? (N18-2)
Copied to clipboard
| Challenge: | a fair comparison of language models is tricky because of the size of the corpora and the variability of orthographic systems. |
| Approach: | They propose a framework for fair cross-linguistic comparison of language models . they show that in some languages, textual expression is harder to predict with n-gram models compared to LSTM models based on translated text . |
| Outcome: | The proposed framework is based on translated text and language models on 21 languages. |
How Persuasive Is Your Context? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Empirically, through aseries of experiments, we show that TPS captures a more nuanced notion of persuasiveness than previously proposed metrics. |
| Approach: | They introduce a targeted persuasion score to quantify how persuasive a given context is to an LM. |
| Outcome: | Empirically, the proposed model captures a more nuanced notion of persuasiveness than previously proposed metrics. |
On the Representational Capacity of Recurrent Neural Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have focused on LMs as formal languages, but they do not consider language membership. |
| Approach: | They extend the Turing completeness result to the probabilistic case . they show that a rationally weighted RLM can simulate any deterministic Turing machine . |
| Outcome: | The proposed model can simulate any deterministic Turing machine with rationally weighted transitions . the proposed model is based on recurrent neural networks with a rational weighting over strings . |
On the Proper Treatment of Units in Surprisal Theory (2026.acl-long)
Copied to clipboard
| Challenge: | empirical work often leaves the notion of a unit underspecified . empirical work has sought to characterize the processing difficulty comprehenders experience . |
| Approach: | They propose a framework for reasoning about surprisal over arbitrary unit inventories . they argue that surprises should be explicit and treat tokenization as implementation detail . |
| Outcome: | The proposed framework disentangles the models' definitions and the regions of interest and treats tokenization as an implementation detail rather than a scientific primitive. |
A Discriminative Latent-Variable Model for Bilingual Lexicon Induction (D18-1)
Copied to clipboard
| Challenge: | Existing methods for bilingual lexicon induction take advantage of word embeddings, but our model is not as efficient as previous work. |
| Approach: | They propose a discriminative latent-variable model for bilingual lexicon induction that combines the bipartite matching dictionary prior and an embedding-based approach. |
| Outcome: | The proposed model outperforms existing models on six language pairs and shows that it mitigates hubness problem. |
Generalized Measures of Anticipation and Responsivity in Online Language Processing (2024.findings-emnlp)
Copied to clipboard
| Challenge: | a generalization of classical information-theoretic measures of predictive uncertainty is proposed for online language processing . entropy and surprisal are two commonly deployed information- theoretic measure of predictive uncertainties in sentence processing based on the probability distribution of upcoming sequences of linguistic units . |
| Approach: | They propose a generalization of classical information-theoretic measures of predictive uncertainty in online language processing based on simulations of incremental linguistic contexts. |
| Outcome: | The proposed generalization of classical measures of predictive uncertainty in online language processing yields a positive effect on reading times and cloze completion probability. |
Classifying Dyads for Militarized Conflict Analysis (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing research examines the origins of militarized conflict by examining bi-lateral relationships between entity pairs and multi-lateral relations among multiple entities. |
| Approach: | They propose to use Wikipedia to model dyadic and systemic causes to compare their correlations with conflict between two entities. |
| Outcome: | The proposed graphs show that Wikipedia articles of allies are semantically more similar than enemies. |
A Probability–Quality Trade-off in Aligned Language Models and its Relation to Sampling Adaptors (2024.emnlp-main)
Copied to clipboard
| Challenge: | a relationship exists between the quality of a string and its probability, p(y), under a language model, and the quality and quality of the string. |
| Approach: | They examine the probability-quality relationship in language models aligned to human preferences through reinforcement learning through human feedback. |
| Outcome: | The proposed method improves the quality of text sampled from a language model by skewing the model towards high-probability strings. |
A Probabilistic Generative Model of Linguistic Typology (N19-1)
Copied to clipboard
| Challenge: | a generative model of languages based on principles-and-parameters posits that languages toggle on or off . linguistic typologists use a set of universal parameters to determine which languages toggle . we show that the correlation between parameters is significant, and that it is not enough to write down the set of parameters available to languages. |
| Approach: | They propose a generative model of language based on exponential-family matrix factorisation. |
| Outcome: | a linguistic model outperforms baseline models on predicting held-out features by exploiting similarities between languages and their features. |
Quantifying the redundancy between prosody and text (2023.emnlp-main)
Copied to clipboard
Lukas Wolf, Tiago Pimentel, Evelina Fedorenko, Ryan Cotterell, Alex Warstadt, Ethan Wilcox, Tamar Regev
| Challenge: | Existing studies suggest partial redundancy between prosody and linguistic information. |
| Approach: | They use large language models to estimate how much information is redundant between prosody and the words themselves. |
| Outcome: | The proposed model can predict prosodic features across prosodic features, including intensity, duration, pauses, and pitch contours. |
Mutual Information Alleviates Hallucinations in Abstractive Summarization (2022.emnlp-main)
Copied to clipboard
| Challenge: | Abstractive summarization models exhibit the tendency to hallucinate, i.e., output content not supported by the source document. |
| Approach: | They propose a decoding strategy that optimizes for pointwise mutual information of source and target tokens when models exhibit uncertainty. |
| Outcome: | The proposed method decreases the probability of hallucinated tokens while maintaining the Rouge and BERT-S scores of top-performing decoding strategies. |
Efficient Computation of Expectations under Spanning Tree Distributions (2021.tacl-1)
Copied to clipboard
| Challenge: | a general framework for inference in spanning tree models is presented . a spanning-tree model is a combinatorial structure that is important for natural language processing . |
| Approach: | They propose a general framework for inference in spanning tree models . they exploit a connection between expectations and gradients, which allows efficient algorithms . |
| Outcome: | The proposed framework is up to 15 times faster than existing algorithms and reduces runtime complexity by a factor of the sentence length. |
Predicting Declension Class from Form and Meaning (2020.acl-main)
Copied to clipboard
| Challenge: | phonological form and meaning of nouns can provide imperfect clues, but they can also be indicative of grammatical gender. |
| Approach: | They propose a method to measure how much information can be gleamed from knowing the form and/or meaning of nouns. |
| Outcome: | The proposed method provides additional quantitative support for a classic linguistic finding that form and meaning are relevant for the classification of nouns into declensions. |
Pointwise Mutual Information as a Performance Gauge for Retrieval-Augmented Generation (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods to improve language models' performance do not exploit this phenomenon . |
| Approach: | They propose to use contextual information to select and construct prompts that improve model performance. |
| Outcome: | The proposed methods show that the mutual information between a context and a question is an effective gauge for language model performance. |
Probing for Reading Times (2026.acl-long)
Copied to clipboard
Eleftheria Tsipidi, Samuel Kiegeland, Francesco Ignazio Re, Tianyang Xu, Mario Giulianelli, Karolina Stanczak, Ryan Cotterell
| Challenge: | a large body of work on probing has demonstrated that language model representations encode a wealth of linguistic information, but it remains unclear whether they also capture cognitive signals about human processing. |
| Approach: | They use regularized linear regression to compare language model representations against scalar predictors. |
| Outcome: | The representations from early layers outperform surprisal in predicting early-pass measures such as first fixation and gaze duration. |
Efficiently Computing Susceptibility to Context in Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | a current language model is able to incorporate information from a user-input context when answering queries, but it is not equally sensitive to subtle changes to that context. |
| Approach: | They propose a metric to quantify the degree to which contexts can influence a model’s response to a query at a distributional level. |
| Outcome: | The proposed method is comparable to Monte Carlo's estimated susceptibility across a diverse set of query domains despite being 70 faster. |
Same Neurons, Different Languages: Probing Morphosyntax in Multilingual Pre-trained Models (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing studies show that multilingual pre-trained models can learn to generalise across languages . however, it remains unclear how these models learn to learn multilingual representations . |
| Approach: | They propose a hypothesis that multilingual pre-trained models can derive language-universal abstractions about grammar by aligning morphosyntactic markers that fulfil a similar grammatical function across languages. |
| Outcome: | The proposed model can derive language-universal abstractions even without explicit supervision. |
BlonDe: An Automatic Evaluation Metric for Document-level Machine Translation (2022.naacl-main)
Copied to clipboard
Yuchen Jiang, Tianyu Liu, Shuming Ma, Dongdong Zhang, Jian Yang, Haoyang Huang, Rico Sennrich, Ryan Cotterell, Mrinmaya Sachan, Ming Zhou
| Challenge: | Standard evaluation metrics, e.g., BLEU, TER and METEOR, focus on the quality of translations at the sentence level and do not consider discourse-level features. |
| Approach: | They propose to use a metric to take discourse coherence into consideration by categorizing discourse-related spans and calculating the similarity-based F1 measure of categorized spans. |
| Outcome: | The proposed metric possesses better selectivity and interpretability at the document-level, and is more sensitive to document- level nuances. |
A Structured Span Selector (2022.naacl-main)
Copied to clipboard
| Challenge: | a typical approach to natural language processing tasks involves selecting text spans and making decisions about them. |
| Approach: | They propose a grammar-based structured span selection model which learns to make use of partial span annotations. |
| Outcome: | The proposed model improves on two popular span prediction tasks. |
Searching for Search Errors in Neural Morphological Inflection (2021.eacl-main)
Copied to clipboard
| Challenge: | Neural sequence-to-sequence models are the predominant choice for language generation tasks. |
| Approach: | They find that on word-level tasks, the empty string is often the global optimum . they suggest that the poor calibration of many neural models may stem from characteristics of a specific subset of tasks rather than general ill-suitedness of such models for language generation. |
| Outcome: | The results suggest that the poor calibration of many neural models may stem from characteristics of a specific subset of tasks rather than general ill-suitedness of such models for language generation. |
A Practical Method for Generating String Counterfactuals (2025.findings-naacl)
Copied to clipboard
| Challenge: | Interventions targeting the representation space of language models (LMs) have emerged as an effective means to influence model behavior. |
| Approach: | They propose a method to convert representation counterfactuals into string counterf actuals and analyze the linguistic alterations corresponding to the intervention. |
| Outcome: | The proposed method analyzes linguistic alterations corresponding to a given representation space intervention and interprets features utilized to encode a specific concept. |
Differentiable Subset Pruning of Transformer Heads (2021.tacl-1)
Copied to clipboard
| Challenge: | Recent work shows that a large proportion of the heads in a Transformer’s multi-head attention mechanism can be safely pruned away without significantly harming the performance of the model. |
| Approach: | They propose a method that prunes a Transformer's multi-head attention mechanism away without significantly harming its performance. |
| Outcome: | The proposed method improves on natural language inference and machine translation tasks while offering precise control of sparsity level. |
A surprisal–duration trade-off across and within the world’s languages (2021.emnlp-main)
Copied to clipboard
| Challenge: | Throughout human evolution, countless languages have evolved, each with unique features. |
| Approach: | They analysed a corpus of 600 languages to find strong evidence for a surprisal–duration trade-off between languages and languages. |
| Outcome: | The proposed model shows that phones are produced faster in languages where they are less surprising and vice versa. |
Examining Gender Bias in Languages with Grammatical Gender (D19-1)
Copied to clipboard
| Challenge: | Existing studies on gender bias in word embeddings focus on English . however, these studies cannot be extended to languages with morphological agreement on gender . |
| Approach: | They propose new metrics to evaluate gender bias in word embeddings of English and Spanish . they extend existing approaches to mitigate gender bias while preserving original embeddables . |
| Outcome: | The proposed methods reduce gender bias while preserving the original embeddings. |
A Bayesian Framework for Information-Theoretic Probing (2021.emnlp-main)
Copied to clipboard
| Challenge: | a recent paper suggests that probing should be seen as approximating a mutual information. |
| Approach: | They propose a Bayesian mutual information framework that probes probing representations from the perspective of Bayes' agents. |
| Outcome: | The proposed framework allows for more intuitive results in scenarios with finite data. |
Benchmarking Compositionality with Formal Languages (2022.coling-1)
Copied to clipboard
| Challenge: | Compositionality is a hallmark of human language, but it is not yet fully understood . recombining known primitive concepts into larger novel combinations is elusive . |
| Approach: | They use finite-state transducers to make a dataset with controllable compositionality . they find that the models either learn the relations completely or not at all . |
| Outcome: | The proposed model learns the relation completely or not at all on large datasets. |
Language Model Evaluation Beyond Perplexity (2021.acl-long)
Copied to clipboard
| Challenge: | a nascent literature on probing language models has focused on studying linguistic phenomena. |
| Approach: | They propose a framework for evaluating the fit of language models to natural language tendencies. |
| Outcome: | The proposed framework evaluates language models to the tendencies of natural language . it shows that the models learn only a subset of the tendancies considered . |
Direct Preference Optimization with an Offset (2024.findings-acl)
Copied to clipboard
| Challenge: | Direct preference optimization (DPO) fine-tunes language models with human preferences . but not all preference pairs are equal; sometimes, the preferred response is only slightly better than the dispreferred one. |
| Approach: | They propose a generalization of direct preference optimization that does not treat every preference pair equally during fine-tuning. |
| Outcome: | The proposed algorithm outperforms DPO on tasks with limited preference pairs . it requires the difference between likelihood of preferred and dispreferred response to be greater than offset value . |
Conditional Poisson Stochastic Beams (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for beam search are based on a deterministic approach, but the results are not as accurate as those used in SBS. |
| Approach: | They propose a method that turns beam search into a stochastic process by using conditional Poisson sampling design instead of taking the maximizing set at each iteration. |
| Outcome: | The proposed method produces lower variance and more efficient estimators than SBS, even showing improvements in high entropy settings. |
Transformers Can Represent n-gram Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing work studies the transformer architecture in terms of language acceptance . n-gram LMs can represent any ngram lm, giving a concrete lower bound on their probabilistic representational capacity. |
| Approach: | They propose to use transformer LMs to represent n-gram LM, a class of language models . they show transformer lms can exactly represent any ngram lmm, giving them a lower bound . |
| Outcome: | The proposed model can represent any n-gram LM, giving a lower bound on their probabilistic representational capacity. |
Tokenization and the Noiseless Channel (2023.acl-long)
Copied to clipboard
| Challenge: | Subword tokenization is a key part of most NLP pipelines, but little is known about why some combinations lead to improved downstream model performance. |
| Approach: | They propose that good tokenizers lead to efficient channel usage . they propose that an optimal encoding assigns extremely long codes to low-frequency subwords . |
| Outcome: | The proposed tokenizers have a very strong correlation with BLEU in machine translation . the proposed function can be used to improve model performance in the downstream task . |
Examining the Inductive Bias of Neural Language Models with Artificial Languages (2021.acl-long)
Copied to clipboard
| Challenge: | a novel method for investigating inductive biases of language models using artificial languages is proposed . we show that modern neural architectures used for language modeling are intrinsically black boxes . |
| Approach: | They propose a method to investigate inductive biases of language models using artificial languages . they use languages to create parallel corpora across languages that differ only in word order . |
| Outcome: | The proposed method shows that language models can be used to model a wide variety of languages. |
Information-Theoretic Probing for Linguistic Structure (2020.acl-main)
Copied to clipboard
| Challenge: | Neural networks are the backbone of modern stateof-the-art natural language processing systems. |
| Approach: | They propose an information-theoretic operationalization of probing as estimating mutual information that contradicts received wisdom . they evaluate on a set of ten typologically diverse languages often underrepresented in NLP research—plus English—totalling eleven languages. |
| Outcome: | The proposed model outperforms existing models on ten typologically diverse languages and English on 11 languages. |
The Paradigm Discovery Problem (2020.acl-main)
Copied to clipboard
| Challenge: | a paradigm discovery problem is a task of learning an inflectional morphological system from unannotated sentences. |
| Approach: | They formalize the paradigm discovery problem and develop evaluation metrics for judging systems . they use word embeddings and string similarity to cluster forms by cell and by paradigm . |
| Outcome: | The proposed system suggests clustering by cell across different inflection classes is the most pressing challenge for future work. |
The Architectural Bottleneck Principle (2022.emnlp-main)
Copied to clipboard
| Challenge: | a recent study examined how much information a model's representations contain . a new approach to probing is to look exactly like the component . |
| Approach: | They propose a new probing principle that aims to estimate how much information a model could extract from its representations. |
| Outcome: | The proposed probes extract syntactic information from the representations of a neural network . the proposed probe is based on the architectural bottleneck principle . |
UniMorph 4.0: Universal Morphology (2022.lrec-1)
Copied to clipboard
Khuyagbaatar Batsuren, Omer Goldman, Salam Khalifa, Nizar Habash, Witold Kieraś, Gábor Bella, Brian Leonard, Garrett Nicolai, Kyle Gorman, Yustinus Ghanggo Ate, Maria Ryskina, Sabrina Mielke, Elena Budianskaya, Charbel El-Khaissi, Tiago Pimentel, Michael Gasser, William Abbott Lane, Mohit Raj, Matt Coler, Jaime Rafael Montoya Samame, Delio Siticonatzi Camaiteri, Esaú Zumaeta Rojas, Didier López Francis, Arturo Oncevay, Juan López Bautista, Gema Celeste Silva Villegas, Lucas Torroba Hennigen, Adam Ek, David Guriel, Peter Dirix, Jean-Philippe Bernardy, Andrey Scherbakov, Aziyana Bayyr-ool, Antonios Anastasopoulos, Roberto Zariquiey, Karina Sheifer, Sofya Ganieva, Hilaria Cruz, Ritván Karahóǧa, Stella Markantonatou, George Pavlidis, Matvey Plugaryov, Elena Klyachko, Ali Salehi, Candy Angulo, Jatayu Baxi, Andrew Krizhanovsky, Natalia Krizhanovskaya, Elizabeth Salesky, Clara Vania, Sardana Ivanova, Jennifer White, Rowan Hall Maudslay, Josef Valvoda, Ran Zmigrod, Paula Czarnowska, Irene Nikkarinen, Aelita Salchak, Brijesh Bhatt, Christopher Straughn, Zoey Liu, Jonathan North Washington, Yuval Pinter, Duygu Ataman, Marcin Wolinski, Totok Suhardijanto, Anna Yablonskaya, Niklas Stoehr, Hossep Dolatian, Zahroh Nuriah, Shyam Ratan, Francis M. Tyers, Edoardo M. Ponti, Grant Aiton, Aryaman Arora, Richard J. Hatcher, Ritesh Kumar, Jeremiah Young, Daria Rodionova, Anastasia Yemelina, Taras Andrushko, Igor Marchenko, Polina Mashkovtseva, Alexandra Serova, Emily Prud’hommeaux, Maria Nepomniashchaya, Fausto Giunchiglia, Eleanor Chodroff, Mans Hulden, Miikka Silfverberg, Arya D. McCarthy, David Yarowsky, Ryan Cotterell, Reut Tsarfaty, Ekaterina Vylomova
| Challenge: | The Universal Morphology project provides broad-coverage instantiated morphological inflection tables for hundreds of diverse languages. |
| Approach: | They propose a language-independent feature schema for rich morphological annotation and a type-level resource of annotated data in diverse languages realizing that schema. |
| Outcome: | The proposed schema has added 66 new languages, including 24 endangered languages. |
The Role of n-gram Smoothing in the Age of Neural Networks (2024.naacl-long)
Copied to clipboard
| Challenge: | n-gram smoothing techniques were used to overcome overfitting problems in neural language models for decades. |
| Approach: | They propose to convert any n-gram smoothing technique into a regularizer compatible with neural language models. |
| Outcome: | The proposed regularizers outperform label smoothing on language modeling and machine translation. |
Applying the Transformer to Character-level Transduction (2021.eacl-main)
Copied to clipboard
| Challenge: | morphological inflection generation and historical text normalization tasks are character-level tasks that outperform recurrent models. |
| Approach: | They propose a technique to handle feature-guided character-level transduction that further improves performance. |
| Outcome: | The transformer outperforms recurrent models on morphological inflection and historical text normalization tasks. |
Sentiment as an Ordinal Latent Variable (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing dictionaries are limited in coverage and sentiment scales vary widely; some are discrete others continuous. |
| Approach: | They propose a Bayesian generative model that learns a composite sentiment dictionary as an interpolation between six existing dictionaries with different scales. |
| Outcome: | The proposed model learns a composite sentiment dictionary as an interpolation between six existing dictionaries with different scales. |
Morphological Irregularity Correlates with Frequency (P19-1)
Copied to clipboard
| Challenge: | a new measure of irregularity is proposed for inflectional morphology of languages . the measure is based on the predictability of forms in a language . |
| Approach: | They propose a measure of irregularity based on the predictability of forms in a language. |
| Outcome: | The proposed measure is the first of its breadth and confirms longstanding proposals from the linguistics literature. |
What Do Language Models Learn in Context? The Structured Task Hypothesis. (2024.acl-long)
Copied to clipboard
| Challenge: | Pre-trained large language models have exhibited an impressive ability to learn in context across various domains, e.g., code generation, education, medicine and even medicine. |
| Approach: | They taxonomize existing candidate theories into three competing hypotheses that explain LLMs’ ability to learn in context. |
| Outcome: | The proposed model can learn a task from in-context examples presented in a demonstration and generalize it to the prompt. |
An Exploration of Left-Corner Transformations (2023.emnlp-main)
Copied to clipboard
| Challenge: | Fig. 1 shows how left-corner transformations remove left recursion from context-free grammars . a common use case is to remove left-recursed grammars, which is necessary for top-down parsing . |
| Approach: | They generalize left-corner transformations to support semiring-weighted production rules . they also provide finer-grained control over which left corners may be moved . |
| Outcome: | The proposed transformation removes left recursion from context-free grammars . it also provides finer-grained control over which left corners may be moved . |
Intrinsic Probing through Dimension Selection (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing research on probing for linguistic structure in word embeddings has focused on intrinsic probing, but what these representations encode about linguistic structures remains unclear. |
| Approach: | They propose a framework that allows us to determine whether linguistic information in word embeddings is dispersed or focal. |
| Outcome: | The proposed framework allows us to determine whether linguistic information in word embeddings is dispersed or focal. |
What About the Precedent: An Information-Theoretic Analysis of Common Law (2021.naacl-main)
Copied to clipboard
| Challenge: | In common law, the outcome of a new case is determined mostly by precedent cases, rather than by existing statutes. |
| Approach: | They propose to model the argumentation of precedent cases and compare them to a case out-come classification task to determine how the precedent influences the outcome of a new case. |
| Outcome: | The proposed method compared arguments of two longstanding jurisprudential views on the European Court of Human Rights (ECtHR) and the precedent cases. |
Generating Text from Language Models (2023.acl-tutorials)
Copied to clipboard
| Challenge: | a growing percentage of natural language processing tasks focus on the generation of text from probabilistic language models. |
| Approach: | They will provide a centralized discussion of critical considerations when choosing how to generate from a language model. |
| Outcome: | This tutorial will provide a centralized discussion of critical considerations when choosing how to generate from a language model. |
Investigating Cross-Linguistic Adjective Ordering Tendencies with a Latent-Variable Model (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing models of crosslinguistic adjective ordering have relied on native speakers' intuitive judgment, not corpus data. |
| Approach: | They propose a latent-variable model that can order adjectives across 24 languages . they use tools and techniques to find universal, cross-linguistic, hierarchical ordering tendencies . |
| Outcome: | The proposed model can order adjectives across 24 languages even when languages are different . similar ordering preferences have been found to apply universally across languages . |
Towards Zero-shot Language Modeling (D19-1)
Copied to clipboard
| Challenge: | a number of natural questions have been asked about the inductive biases of neural networks on core NLP tasks. |
| Approach: | They construct an informative prior for held-out languages on a task of character-level, open-vocabulary language modelling. |
| Outcome: | The proposed model outperforms baseline models with an uninformative prior in both zero-shot and few-shot settings, showing that it is imbued with universal linguistic knowledge. |
Computational Expressivity of Neural Language Models (2024.acl-tutorials)
Copied to clipboard
| Challenge: | Language models (LMs) are at the forefront of NLP research due to their versatility across diverse tasks. |
| Approach: | This tutorial will provide a framework for formal analysis of modern language models using tools from formal language theory. |
| Outcome: | This tutorial will provide a framework for formal analysis of modern language models using tools from formal language theory (FLT). |
On Homophony and Rényi Entropy (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent theories of language optimality have tried to justify its prevalence, arguing that homophony enables the reuse of efficient wordforms and is thus beneficial for languages. |
| Approach: | They propose a new information-theoretic quantification of a language’s homophony: the sample Rényi entropy. |
| Outcome: | The proposed method is more nuanced than either Piantadosi et al.'s or Trott and Bergen's results. |
Generalizing Backpropagation for Gradient-Based Interpretability (2023.acl-long)
Copied to clipboard
| Challenge: | Several feature-attribution methods for interpreting deep neural networks rely on computing the gradients of a model’s output with respect to its inputs, but they reveal little about the inner workings of the model itself. |
| Approach: | They propose a generalized backpropagation algorithm that generalizes the gradient computation of a model to efficiently compute other interpretable statistics about the gradient graph of neural networks. |
| Outcome: | The proposed generalized algorithm can be used to compute other interpretable statistics about the gradient graph of a neural network, such as the highest-weighted path and entropy. |
Higher-order Derivatives of Weighted Finite-state Machines (2021.acl-short)
Copied to clipboard
| Challenge: | Weighted finite-state machines (WFSMs) have a storied role in NLP . e.g., conditional random fields for part-of-speech tagging are considered special cases of WFSM. |
| Approach: | They propose an algorithm for evaluating derivatives of all orders that is faster than prior algorithms . they also find that it is faster for computing second-order expectations . |
| Outcome: | The proposed algorithm is faster than prior algorithms and can compute second-order expectations. |
Hard Non-Monotonic Attention for Character-Level Transduction (D18-1)
Copied to clipboard
| Challenge: | Character-level string-to-string transductions are an important component of NLP tasks . hard non-monotonic attention models have been used for sequence modeling tasks involving characters . |
| Approach: | They propose an exact algorithm for marginalizing over the exponential number of non-monotonic alignments between two strings. |
| Outcome: | The proposed algorithm outperforms soft attention and improves performance over stochastic approximation. |
Is Sparse Attention more Interpretable? (2021.acl-short)
Copied to clipboard
| Challenge: | Sparse attention has been claimed to increase model interpretability . however, the attention distribution is typically over representations internal to the model rather than the inputs themselves . |
| Approach: | They conduct experiments to understand how sparsity affects our ability to use attention as an explainability tool. |
| Outcome: | The proposed model does not map to a sparse set of influential inputs, but rather to fewer inputs. |
Parameter Space Factorization for Zero-Shot Learning across Tasks and Languages (2021.tacl-1)
Copied to clipboard
| Challenge: | Currently, there are only 24 languages in the world that have not been annotated . transferring knowledge across domains is a common solution . |
| Approach: | They propose a Bayesian generative model for the space of neural parameters that factorizes into latent variables for each language and each task. |
| Outcome: | The proposed model can perform better than state-of-the-art methods with a typologically diverse sample of 33 languages from 4 continents and 11 families. |
Contextualization of Morphological Inflection (N19-1)
Copied to clipboard
| Challenge: | In this paper, we isolate the task of predicting a fully inflected sentence from its partially lemmatized version. |
| Approach: | They propose a task that requires morphological features to be inferred from sentential context . they propose morphology-based models that explicitly reconstruct morphologic features before predicting inflected forms . |
| Outcome: | The proposed model is able to predict inflected sentences without relying on morphological annotations. |
Efficient Semiring-Weighted Earley Parsing (2023.acl-long)
Copied to clipboard
| Challenge: | Using Earley's context-free parsing algorithm, we show that the speed-ups are effective in practice. |
| Approach: | They propose a context-free parsing algorithm with various known and new speed-ups that improve Earley's (1970) O(N3|G||R|) They also propose 'a binarized version' that achieves runtime of O(M| |G| when the grammar is represented compactly as a single finite-state automaton M. |
| Outcome: | The proposed algorithm can be used to reduce the complexity of CKY on a binarized version of the grammar G. |
Unsupervised Discovery of Gendered Language through Latent-Variable Modeling (P19-1)
Copied to clipboard
| Challenge: | a recent study has focused on the ways in which language is gendered . positive adjectives used to describe women are more often related to their bodies . |
| Approach: | They propose a model that models adjective choice and its sentiment given the natural gender of a head noun. |
| Outcome: | The proposed model shows that positive adjectives used to describe women are more often related to their bodies than positive adjective words used to explain men. |
On the Representational Capacity of Neural Language Models with Chain-of-Thought Reasoning (2024.acl-long)
Copied to clipboard
| Challenge: | Existing theoretical treatments have shed light on some aspects of CoT, but we are still far from a concrete understanding of the concept. |
| Approach: | They propose to formalize CoT reasoning in a probabilistic setting to bridge this gap . they show that LMs can represent the same family of distributions over strings as probabilistic Turing machines. |
| Outcome: | The proposed model can represent the same family of distributions over strings as Turing machines. |
Probing via Prompting (2022.naacl-main)
Copied to clipboard
| Challenge: | Pre-trained language models have increased the performance of data-driven natural language processing (NLP) models on a wide variety of tasks. |
| Approach: | They propose a model-free approach to probing via prompting which formulates probing as a prompting task and combine pruning to analyze where the model stores the linguistic information in its architecture. |
| Outcome: | The proposed approach extracts information from pre-trained models while learning much less on its own. |
UniMorph 3.0: Universal Morphology (2020.lrec-1)
Copied to clipboard
Arya D. McCarthy, Christo Kirov, Matteo Grella, Amrit Nidhi, Patrick Xia, Kyle Gorman, Ekaterina Vylomova, Sabrina J. Mielke, Garrett Nicolai, Miikka Silfverberg, Timofey Arkhangelskiy, Nataly Krizhanovsky, Andrew Krizhanovsky, Elena Klyachko, Alexey Sorokin, John Mansfield, Valts Ernštreits, Yuval Pinter, Cassandra L. Jacobs, Ryan Cotterell, Mans Hulden, David Yarowsky
| Challenge: | Explicit modeling of morphology has demonstrable benefits for language modeling, speech recognition, word embedding and keyword search. |
| Approach: | They propose a language-independent feature schema for rich morphological annotation and a type-level resource for annotated data in diverse languages. |
| Outcome: | The proposed schema has been improved to make it more complete and correct, and adds 66 new languages and parts of speech for 12 languages. |
Algorithms for Weighted Pushdown Automata (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing dynamic programming algorithms for PDAs often resort to a PDA-to-CFG conversion. |
| Approach: | They propose to use a pushdown automaton to reduce the space requirements by a factor of |Gamma| or the runtime by reducing the number of states. |
| Outcome: | The proposed algorithms reduce the space requirements by a factor of |Gamma| or reduce the runtime by fewer states. |
A Non-Linear Structural Probe (2021.naacl-main)
Copied to clipboard
| Challenge: | Probing is a method of investigating the encoding of knowledge in contextual representations. |
| Approach: | They propose to kernelize a metric and develop a non-linear variant with an identical number of parameters by using a kernel-based probe. |
| Outcome: | The proposed probe learns only linear transformations and achieves statistically significant performance improvement over baseline in all languages. |
Don’t Forget the Long Tail! A Comprehensive Analysis of Morphological Generalization in Bilingual Lexicon Induction (D19-1)
Copied to clipboard
| Challenge: | Human translators have to translate rare inflections due to Zipfian distribution of words in a language. |
| Approach: | They introduce 40 morphologically complete dictionaries in 10 languages and evaluate three of the best performing models on the task of translation of less frequent morphology. |
| Outcome: | The proposed models perform better on infrequent morphological inflections and add a simple constraint at training time. |
Generalized Entropy Regularization or: There’s Nothing Special about Label Smoothing (2020.acl-main)
Copied to clipboard
| Challenge: | Prior work has explored regularizing the output distributions of probabilistic models to alleviate overfitting. |
| Approach: | They propose a family of entropy regularizers that have a connection to regularization . they find that label smoothing provably does not allow for sparsity in an output distribution . |
| Outcome: | The proposed method improves the relationship between model entropy and performance on language generation tasks. |
On the Efficacy of Sampling Adapters (2023.acl-long)
Copied to clipboard
| Challenge: | Using sampling adapters can improve the quality of the generated text. |
| Approach: | They propose a framework for understanding sampling adapters and propose 'sampling adapters' they argue that the shift enforced by them can be viewed as a trade-off between precision and recall . |
| Outcome: | The proposed framework can be used to improve the quality of language models by modifying their distributions to improve their precision and recall. |
A Cognitive Regularizer for Language Modeling (2021.acl-long)
Copied to clipboard
| Challenge: | a uniform information density hypothesis is used to explain certain linguistic phenomena . a regularizer that encodes the UID hypothesis can be used for language training . |
| Approach: | They propose to augment the canonical MLE objective with a regularizer that encodes UID . they find that regularization consistently improves perplexity in language models . |
| Outcome: | The proposed hypothesis can be operationalized as an inductive bias for language modeling. |
Speakers Fill Lexical Semantic Gaps with Context (2020.emnlp-main)
Copied to clipboard
| Challenge: | Lexical ambiguity is widespread in language, allowing for the reuse of economical word forms and thus making language more efficient. |
| Approach: | They propose two ways to estimate lexical ambiguity as the entropy of meanings a word can take . they validate this hypothesis by using WordNet and BERT . |
| Outcome: | The proposed method shows that on six high-resource languages, there are significant correlations between the estimate and the number of synonyms a word has in WordNet. |
Recurrent Neural Language Models as Probabilistic Finite-state Automata (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have focused on the expressive power of recurrent neural network LMs to recognize unweighted formal languages. |
| Approach: | They propose to model a strict subset of probabilistic finite-state automata with RNNs . they show that an RNN requires left(N ||right) neurons to represent an LM . |
| Outcome: | The proposed language models can represent a strict subset of probabilistic distributions expressed by finite-state models. |
Combining Sentiment Lexica with a Multi-View Variational Autoencoder (N19-1)
Copied to clipboard
| Challenge: | a new model of sentiment lexica is being developed to combine disparate scales into a common representation. |
| Approach: | They propose a model that unifies disparate scales into a common latent representation . they evaluate a text classification task using nine English-Language sentiment datasets . |
| Outcome: | The proposed model outperforms six individual sentiment lexica and a simple combination thereof. |
Analyzing Wrap-Up Effects through an Information-Theoretic Lens (2022.acl-short)
Copied to clipboard
| Challenge: | a lack of studies targeting naturalistic sentence-final reading behavior is likely to explain the lack of data on reading time (RT) data is omitted due to the confounding factors introduced by so-called "wrap-up effects" |
| Approach: | They propose to look for a link between “wrap-up effects” and information theoretic quantities such as word and context information content. |
| Outcome: | The proposed model omits data on words at the end of sentences or clauses to control for the confounding factors introduced by wrap-up effects. |
Quantifying the Semantic Core of Gender Systems (D19-1)
Copied to clipboard
| Challenge: | a large number of languages employ grammatical gender on the lexeme, but is it truly arbitrary? a recent study shows that the relationship between grammamatical gender and lexical semantics is opaque. |
| Approach: | They propose a method to correlating inanimate nouns' gender with lexical semantics . they find that the gender systems of 18 languages exhibit a significant correlation with a definition . |
| Outcome: | a new study shows that the gender assignments of 18 languages are arbitrary . the authors show that the correlation between gender and semantics is significant . |
It’s Easier to Translate out of English than into it: Measuring Neural Translation Difficulty by Cross-Mutual Information (2020.acl-main)
Copied to clipboard
| Challenge: | Current state-of-the-art MT systems are based on neural networks, but it is unclear whether all translation directions are equally easy (or hard) to model for NMT. |
| Approach: | They propose an asymmetric information-theoretic metric of machine translation difficulty that exploits the probabilistic nature of most neural machine translation models. |
| Outcome: | The proposed metric allows us to better evaluate the difficulty of translating text into the target language while controlling for the difficulty independent of the translation task. |