Papers by Kyle Mahowald
Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionality (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent visuolinguistic pre-trained models fail miserably on the Winoground dataset, which challenges models to match paired images and English captions. |
| Approach: | They propose to annotate a Winoground dataset that challenges visuolinguistic models to match paired images and English captions with items constructed to overlap lexically but differ in meaning. |
| Outcome: | The proposed dataset challenges models to match paired images and English captions with items constructed to overlap lexically but differ in meaning. |
Causal Interventions Reveal Shared Structure Across English Filler–Gap Constructions (2025.emnlp-main)
Copied to clipboard
| Challenge: | Language Models (LMs) have emerged as powerful sources of evidence for linguists seeking to develop theories of syntax. |
| Approach: | They propose to use causal interpretability methods to characterize abstract mechanisms that LMs learn to use by transferring a wh-filler-gap structure into a gap-less c++ class. |
| Outcome: | The proposed methods can characterize the abstract mechanisms that LMs learn to use, and challenge claims that they can be learned only with strong innate priors. |
Convergence and Divergence of Language Models under Different Random Seeds (2025.emnlp-main)
Copied to clipboard
| Challenge: | a large body of work has examined the training dynamics of language models. |
| Approach: | They investigate the convergence of language models (LMs) trained under different random seeds . they find that larger models reconverge faster in later training stages, while smaller models never actually reconverge. |
| Outcome: | The proposed model size and training checkpoints influence convergence of language models under different seeds. |
Revisiting the Optimality of Word Lengths (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing theories of communication cost are based on the length of an utterance, but they are not based upon the frequency of the utterant. |
| Approach: | They propose a method to minimize CCH's cost by comparing language lengths to surprisal's expectation and variance-to-mean ratios. |
| Outcome: | The proposed method does not minimize CCH’s cost, but rather a lower bound, which we term CCH-lower. |
How (Non-)Optimal is the Lexicon? (2021.naacl-main)
Copied to clipboard
| Challenge: | lexical meanings are mapped to wordforms by usage pressures and constraints on sequences of symbols. |
| Approach: | They propose a coding-theoretic view of the lexicon and a novel generative statistical model to quantify its compressibility under various constraints. |
| Outcome: | The proposed model shows that (compositional) morphology and graphotactics can account for most of the complexity of natural codes—as measured by code length. |
A Massively Multilingual Analysis of Cross-linguality in Shared Embedding Space (2021.emnlp-main)
Copied to clipboard
| Challenge: | Cross-lingual language models house representations for many different languages in the same space. |
| Approach: | They investigate linguistic and non-linguistic factors affecting sentence-level alignment in cross-lingual pretrained language models for 101 languages and 5,050 language pairs. |
| Outcome: | The results show that word order agreement and agreement in morphological complexity are strongest predictors of cross-linguality. |
Counterfactual Probing for the Influence of Affect and Specificity on Intergroup Bias (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing work on bias in NLP only considers negative or pejorative language use. |
| Approach: | They propose a revised framing of bias in terms of intergroup social context and its effects on language output. |
| Outcome: | The proposed framework is based on a model of intergroup relationships in English language tweets. |
What do tokens know about their characters and how do they know it? (2022.naacl-main)
Copied to clipboard
| Challenge: | Pre-trained language models that use subword tokenization schemes can succeed at a variety of language tasks that require character-level information. |
| Approach: | They propose to use word tokenization schemes to probe what word pieces encode . they show that larger models can encode character-level information . |
| Outcome: | The proposed models can encode character-level information and perform better on non-Latin alphabets. |
Is It JUST Semantics? A Case Study of Discourse Particle Understanding in LLMs (2025.findings-acl)
Copied to clipboard
William Berkeley Sheffield, Kanishka Misra, Valentina Pyatkin, Ashwini Deo, Kyle Mahowald, Junyi Jessy Li
| Challenge: | Discourse particles are crucial elements that subtly shape the meaning of text. |
| Approach: | They examine the capacity of linguists to distinguish fine-grained senses of English *just* . they find that they struggle to fully capture more subtle nuances of discourse particles . |
| Outcome: | The study shows that linguists struggle to capture subtle nuances of discourse particles. |
With Little Power Comes Great Responsibility (2020.emnlp-main)
Copied to clipboard
| Challenge: | Underpowered experiments make it more difficult to discern the difference between statistical noise and meaningful model improvements and increase the chances of exaggerated findings. |
| Approach: | They characterize typical statistical power for a variety of settings and characterize it by a set of existing NLP papers and datasets. |
| Outcome: | The authors characterize typical power for a variety of settings and find it common in the literature. |
For Generated Text, Is NLI-Neutral Text the Best Text? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a perfectly informative agent would eschew utterances that are redundant or contradict that which they have already said. |
| Approach: | They propose to use a pre-trained NLI model to assess whether a sentence entails, contradicts, or is neutral to prompt and preceding text. |
| Outcome: | The proposed model is predictive of generation errors made by GPT-3. |
What Can String Probability Tell Us About Grammaticality? (2026.tacl-1)
Copied to clipboard
| Challenge: | linguistic theories have argued that language models have largely achieved grammatical competence, but they will assign non-zero probability to all strings. |
| Approach: | They propose a theoretical framework for analyzing string probabilities in linguistics based on simple assumptions about the generative process of corpus data. |
| Outcome: | The proposed framework makes three predictions using 280K sentence pairs in English and Chinese. |
Deep Subjecthood: Higher-Order Grammatical Features in Multilingual BERT (2021.eacl-main)
Copied to clipboard
| Challenge: | a recent study has shown that multilingual BERT encodes sentences in structurally meaningful ways. |
| Approach: | They analyze how morphosyntactic alignment manifests across embedding spaces of languages . they train classifiers to recover subjecthood of mBERT embedds in transitive sentences . |
| Outcome: | The proposed model encodes a high-order grammatical feature of morphosyntactic alignment across languages . the results show that the classifier distributions reflect the morphological alignment of their training languages based on the results . |
Constructions are Revealed in Word Distributions (2025.emnlp-main)
Copied to clipboard
| Challenge: | Construction grammar posits that constructions are form-meaning pairings that are acquired through experience with language. |
| Approach: | They propose to use a RoBERTa model to model constructions as patterns of statistical affinity . they show that statistical affinity is likely an important, but partial, signal available to learners . |
| Outcome: | The proposed model shows that constructions will be revealed as patterns of statistical affinity . the proposed model is based on a model that is able to distinguish constructions from text . |
Inducing Character-level Structure in Subword-based Language Models with Type-level Interchange Intervention Training (2023.findings-acl)
Copied to clipboard
| Challenge: | Language models operating on subword units are challenging for character-level manipulations, authors say . authors develop a framework to learn robust character representations inside subword-based models . |
| Approach: | They propose a causal intervention framework to learn robust character representations inside subword-based language models. |
| Outcome: | The proposed model outperforms character-level models on more complex tasks . it improves robustness on unseen token sequences and leads to human-interpretable representations of characters. |
When classifying grammatical role, BERT doesn’t care about word order... except when it matters (2022.acl-short)
Copied to clipboard
| Challenge: | Recent work has shown large language models are surprisingly word order invariant . however, word order knowledge is crucial in defining later-layer representations of words . |
| Approach: | They probe grammatical role representations in English BERT and GPT-2 to find word order crucial . they find word orders are crucial in defining later-layer representations of words in non-prototypical positions . |
| Outcome: | The proposed model is based on natural prototypical inputs where word order is crucial for correct classification. |
Experimental Contexts Can Facilitate Robust Semantic Property Inference in Language Models, but Inconsistently (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent zero-shot evaluations have highlighted important limitations in the abilities of language models (LMs) to perform meaning extraction. |
| Approach: | They propose to use in-context examples and instructions to improve LMs' robustness in performing property inheritance. |
| Outcome: | The proposed model can perform non-trivial property inheritance on in-context examples and instructions, but it is inconsistent with the task. |
Mechanisms of Prompt-Induced Hallucination in Vision–Language Models (2026.acl-long)
Copied to clipboard
William Rudman, Michal Golovanevsky, Dana Arad, Yonatan Belinkov, Carsten Eickhoff, Ritambhara Singh, Kyle Mahowald
| Challenge: | Large vision–language models (VLMs) often hallucinate by favoring textual prompts over visual evidence. |
| Approach: | They study the failure mode of large vision–language models by focusing on textual prompts over visual evidence. |
| Outcome: | The proposed model overestimates the number of objects in an image . it hallucinates additional waterlilies when asked to describe a mismatched number of items . the model ablation reduces prompt-induced hallucinosities by at least 40% without additional training . |
Do *they* mean ‘us’? Interpreting Referring Expression variation under Intergroup Bias (2024.findings-emnlp)
Copied to clipboard
| Challenge: | We model intergroup bias as a tagging task on English sports comments from forums dedicated to fandom for NFL teams . linguistic descriptions of win probability are used for large-scale analysis of intergroup variation . |
| Approach: | They propose to model intergroup bias as a tagging task on NFL fan comments . they use linguistic models to model the bias and use them to generate large-scale annotations . |
| Outcome: | The proposed model can reveal unobserved variations in the form of referents across win probabilities. |
Language Models Learn Rare Phenomena from Less Rare Phenomena: The Case of the Missing AANNs (2024.emnlp-main)
Copied to clipboard
| Challenge: | Language models learn rare syntactic phenomena by generalization vs. memorization, a study finds . aannalysis experiments show that humans learn rare grammatical structures by generalizing from less rare phenomena. |
| Approach: | They iteratively trained transformer language models on a systematically manipulated corpus and evaluated their learning of a rare grammatical phenomenon. |
| Outcome: | The results show that language models learn rare grammatical phenomena by generalization vs. memorization . human-scale corpora are used to train the models and compare their learning to counterfactual corpors . |
Elaborative Simplification as Implicit Questions Under Discussion (2023.emnlp-main)
Copied to clipboard
| Challenge: | Automated text simplification is often thought of as a monolingual translation task . this view fails to account for elaborative simplification, where new information is added into the simplified text. |
| Approach: | They propose to view elaborative simplification through the lens of the Question Under Discussion framework . they propose to model 1.3K elongations accompanied by implicit QUDs to investigate what writers elaborate upon . |
| Outcome: | The proposed framework provides a robust way to investigate what writers elaborate upon, how they elaborate, and how elaborations fit into the discourse context. |
A Method for Studying Semantic Construal in Grammatical Constructions with Interpretable Contextual Embedding Spaces (2023.acl-long)
Copied to clipboard
| Challenge: | Existing paradigms for the linguistically oriented exploration of large neural language models include treating the model as a linguistic test subject by measuring output on test sentences and building probing classifiers on top of embeddings to test whether the embeddables are sensitive to certain properties like dependency structure. |
| Approach: | They project contextual embeddings into interpretable semantic spaces, each defined by a different set of psycholinguistic feature norms. |
| Outcome: | The proposed method can probe the distributional meaning of syntactic constructions at a templatic level, abstracted away from specific lexemes. |
A Discerning Several Thousand Judgments: GPT-3 Rates the Article + Adjective + Numeral + Noun Construction (2023.eacl-main)
Copied to clipboard
| Challenge: | a number of studies have focused on the use of rare, idiosyncratic constructions. |
| Approach: | They ask GPT-3 to give acceptability judgments on an English-language construction . they validate the prompt and then zero in on the AANN construction based on CoLA corpus . |
| Outcome: | The proposed judgments are similar to human judgments but differ from the literature and from each other. |