Papers by Adina Williams
Dynatask: A Framework for Creating Dynamic AI Benchmark Tasks (2022.acl-demo)
Copied to clipboard
Tristan Thrush, Kushal Tirumala, Anmol Gupta, Max Bartolo, Pedro Rodriguez, Tariq Kane, William Gaviria Rojas, Peter Mattson, Adina Williams, Douwe Kiela
| Challenge: | Open source system for setting up custom NLP tasks aims to lower technical knowledge and effort required for hosting and evaluating state-of-the-art models. |
| Approach: | They propose to integrate Dynatask with Dynabench to simplify benchmarking . they use a dataset to collect and clean data and train and evaluate models . |
| Outcome: | Dynatask is an open source system for setting up custom NLP tasks . it is integrated with Dynabench, a research platform for rethinking benchmarking in AI . |
Queens are Powerful too: Mitigating Gender Bias in Dialogue Generation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Social biases present in data are often directly reflected in the predictions of models trained on that data. |
| Approach: | They analyze gender bias in dialogue data and propose techniques to mitigate it . they use counterfactual data augmentation, targeted data collection, and bias controlled training . |
| Outcome: | The proposed techniques mitigate gender bias by balancing genderedness of generated dialogue utterances. |
Multi-Dimensional Gender Bias Classification (2020.emnlp-main)
Copied to clipboard
| Challenge: | a novel framework decomposes gender bias in text along several pragmatic and semantic dimensions . language is a primary means by which people communicate, express identities and categorize themselves . unwanted gender biases can affect downstream applications, leading to poor user experiences . |
| Approach: | They propose a framework that decomposes gender bias in text along several dimensions . they annotate eight large scale datasets with gender information and collect a benchmark . |
| Outcome: | The proposed framework decomposes gender bias in text along several pragmatic and semantic dimensions. |
Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESupposition (2020.acl-main)
Copied to clipboard
| Challenge: | Natural language inference (NLI) is an increasingly important task for natural language understanding . however, the ability of NLI models to make pragmatic inferences remains understudied . |
| Approach: | They use semi-automatically generated sentence pairs to evaluate whether NLI models make pragmatic inferences. |
| Outcome: | The proposed model trains on multiNLI and shows that it learns to draw pragmatic inferences. |
Language model acceptability judgements are not always robust to context (2023.acl-long)
Copied to clipboard
Koustuv Sinha, Jon Gauthier, Aaron Mueller, Kanishka Misra, Keren Fuentes, Roger Levy, Adina Williams
| Challenge: | a recent study found that models prefer acceptable inputs over acceptable ones. |
| Approach: | They find that model judgements are generally robust when placed in randomly sampled linguistic contexts, but unstable when contexts match the test stimuli in syntactic structure. |
| Outcome: | The proposed model performance improves when contexts match syntactic structure, and declines when they are unacceptable. |
Improving Model Evaluation using SMART Filtering of Benchmark Datasets (2025.naacl-long)
Copied to clipboard
| Challenge: | Creating high quality human-annotated datasets is difficult due to dataset saturation. |
| Approach: | They propose a method to filter a subset of test examples from existing benchmarks by removing less informative and lower quality examples. |
| Outcome: | The proposed method reduces dataset size by 48% while increasing Pearson correlation with rankings from ChatBot Arena. |
Analyzing Dynamic Adversarial Training Data in the Limit (2022.findings-acl)
Copied to clipboard
| Challenge: | Dynamic adversarial data collection (DADC) can be used to build models that are robust across a wide range of test inputs. |
| Approach: | They propose to run Dynamic adversarial data collection over many rounds to maximize its training-time benefits. |
| Outcome: | The proposed model makes 26% fewer errors on the premise paragraphs compared to models trained on non-adversarial examples. |
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference (N18-1)
Copied to clipboard
| Challenge: | et al., 1996, show that many of the most actively studied problems in NLP depend in large part on natural language understanding (NLU). |
| Approach: | They propose a dataset for machine learning that uses ten different genres of English to evaluate sentences for their meanings. |
| Outcome: | The multi-genre natural language inference corpus is one of the largest available for natural language understanding. |
A Tale of a Probe and a Parser (2020.acl-main)
Copied to clipboard
| Challenge: | researchers train supervised models to extract linguistic structure from output of another model . supervised model can be used to perform tasks such as part-of-speech tags or dependency trees . |
| Approach: | They compare a structural probe to a more traditional parser with a lightweight parameterisation. |
| Outcome: | The structural probe outperforms a traditional parser on seven of nine languages . the researchers found that the model outperformed the parsers by 11.1 points . |
Measuring the Similarity of Grammatical Gender Systems by Comparing Partitions (2020.emnlp-main)
Copied to clipboard
| Challenge: | A grammatical gender system divides a lexicon into a small number of fixed categories with fixed usage across speakers. |
| Approach: | They propose to define gender systems extensionally to reduce comparisons to cluster evaluation by comparing pairwise overlaps between gender systems. |
| Outcome: | The proposed measures are based on a phylogenetic tree over extant Indo-European languages. |
Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little (2021.emnlp-main)
Copied to clipboard
| Challenge: | masked language models (MLMs) pre-train to model higher-order word co-occurrence statistics . authors suggest that such models have learned to represent syntactic structures prevalent in classical NLP pipelines . purely distributional information largely explains the success of pre-training, authors say . |
| Approach: | They propose to pre-train masked language models on sentences with random shuffled word order and show they still achieve high accuracy after fine-tuning on many downstream tasks. |
| Outcome: | The proposed model performs well according to parametric syntactic probes . the authors argue that the model is not all that different from earlier distributional models . |
Calibrating LLM Judges: Linear Probes for Fast and Reliable Uncertainty Estimation (2026.acl-industry)
Copied to clipboard
Bhaktipriya Radharapu, Eshika Saxena, Kenneth Li, Chenxi Whitehouse, Adina Williams, Nicola Cancedda
| Challenge: | Existing methods for obtaining well-calibrated uncertainty estimates are poorly calibrated or computationally expensive. |
| Approach: | They propose a linear probe that provides calibrated uncertainty estimates from reasoning judges’ hidden states, requiring no additional model training. |
| Outcome: | The proposed method achieves superior calibration compared to existing methods with x computational savings, generalizes robustly to unseen evaluation domains, and delivers higher accuracy on high-confidence predictions. |
Are Female Carpenters like Blue Bananas? A Corpus Investigation of Occupation Gender Typicality (2024.findings-acl)
Copied to clipboard
| Challenge: | Typicality effects shape the way we use language, says a researcher . mentioning unusual properties of events is not a typical feature of bananas, he says . |
| Approach: | They explore whether occupations and gender display the same patterns of mentioning as do bananas and color . they find that gender mentioning is correlated with femaleness of occupation in particular . |
| Outcome: | a new study shows that people tend to mention unusual properties of events when they talk about them . the findings contradict the idea that gender mentioning occurs when special events are pointed out . |
On the Relationships Between the Grammatical Genders of Inanimate Nouns and Their Co-Occurring Adjectives and Verbs (2021.tacl-1)
Copied to clipboard
| Challenge: | In many languages, nouns possess grammatical genders. |
| Approach: | They use large-scale corpora and tools from NLP and information theory to test whether there is a relationship between grammatical genders of inanimate nouns and adjectives used to describe them. |
| Outcome: | The results show that there is a statistically significant relationship between the grammatical genders of inanimate nouns and adjectives used to describe them in all six languages. |
Dynabench: Rethinking Benchmarking in NLP (2021.naacl-main)
Copied to clipboard
Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, Adina Williams
| Challenge: | Dynabench is an open-source platform for dynamic dataset creation and model benchmarking. |
| Approach: | They propose an open-source platform for dynamic dataset creation and model benchmarking. |
| Outcome: | The proposed platform can be used to create models that fail on simple challenges and falter in real-world scenarios. |
“I’m sorry to hear that”: Finding New Biases in Language Models with a Holistic Descriptor Dataset (2022.emnlp-main)
Copied to clipboard
| Challenge: | Language models are increasingly important to measure all possible demographic markers of identity . many datasets for measuring bias are limited in their coverage of demographic axes . |
| Approach: | They propose a bias measurement dataset that includes nearly 600 descriptor terms across 13 demographic axes. |
| Outcome: | The proposed dataset explores, detects, and reduces biases in language models. |
On the Idiosyncrasies of the Mandarin Chinese Classifier System (N19-1)
Copied to clipboard
| Challenge: | idiosyncrasies of the Chinese classifier system have been studied, but little work has been done to quantify them with statistical methods. |
| Approach: | They propose an information-theoretic approach to measuring idiosyncrasies in Mandarin Chinese by calculating the mutual information between the distribution over classifiers and distributions over other linguistic quantities. |
| Outcome: | The proposed method reduces uncertainty in Mandarin Chinese classifiers by knowing semantic information about nouns that they modify. |
Pareto Probing: Trading Off Accuracy for Complexity (2020.emnlp-main)
Copied to clipboard
| Challenge: | Neural networks are a pillar of modern NLP systems, but their inner workings are poorly understood. |
| Approach: | They propose a probe metric that reflects the trade-off between probe complexity and performance: the Pareto hypervolume. |
| Outcome: | The proposed probe metric conforms to accepted rankings among contextual representations, and is more complex than other probe tasks. |
EmphAssess : a Prosodic Benchmark on Assessing Emphasis Transfer in Speech-to-Speech Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | EmphAssess evaluates speech-to-speech models' ability to encode and reproduce prosodic emphasis across a change of speaker and language. |
| Approach: | They propose a prosodic benchmark to evaluate the ability of speech-to-speech models to encode and reproduce prosodic emphasis. |
| Outcome: | The proposed model can encode and reproduce prosodic emphasis across speech inputs and outputs . EmphaClass classifies emphasis at the frame or word level . |
Adversarial NLI: A New Benchmark for Natural Language Understanding (2020.acl-main)
Copied to clipboard
| Challenge: | a new large-scale NLI benchmark dataset is presented to test models on a variety of popular NLIs. |
| Approach: | They propose a large-scale NLI benchmark dataset that is iteratively compared with a human-and-model-in-the-loop procedure. |
| Outcome: | The proposed method can be applied in a never-ending learning scenario, becoming a moving target for NLU, rather than a static benchmark that will quickly saturate. |
On the Machine Learning of Ethical Judgments from Natural Language (2022.naacl-main)
Copied to clipboard
| Challenge: | a recent study examines the morality of NLP models that can take in arbitrary text and output a moral judgment . a Delphi project is a popular system for moral prediction, but it has received criticism . |
| Approach: | They propose to critique NLP methods for automating ethical decision-making . they examine a nascent task of predicting moral and ethical decisions from text . |
| Outcome: | The proposed model is unsafe at any accuracy, the authors argue . they argue that the proposed model could be useful in NLP, but not in AI. |
Robustness of Named-Entity Replacements for In-Context Learning (2023.findings-emnlp)
Copied to clipboard
Saeed Goodarzi, Nikhil Kagita, Dennis Minn, Shufan Wang, Roberto Dessi, Shubham Toshniwal, Adina Williams, Jack Lanchantin, Koustuv Sinha
| Challenge: | Modern large language models perform in-context learning, where query- answer demonstrations are shown before the final query. |
| Approach: | They propose to use in-context learning to prompt queries before they are answered . they find that the choice of demonstrations can affect model performance . |
| Outcome: | The proposed model performance improves on named entity replacements across three reasoning tasks and two popular LLMs. |
Predicting Declension Class from Form and Meaning (2020.acl-main)
Copied to clipboard
| Challenge: | phonological form and meaning of nouns can provide imperfect clues, but they can also be indicative of grammatical gender. |
| Approach: | They propose a method to measure how much information can be gleamed from knowing the form and/or meaning of nouns. |
| Outcome: | The proposed method provides additional quantitative support for a classic linguistic finding that form and meaning are relevant for the classification of nouns into declensions. |
Benchmarking Compositionality with Formal Languages (2022.coling-1)
Copied to clipboard
| Challenge: | Compositionality is a hallmark of human language, but it is not yet fully understood . recombining known primitive concepts into larger novel combinations is elusive . |
| Approach: | They use finite-state transducers to make a dataset with controllable compositionality . they find that the models either learn the relations completely or not at all . |
| Outcome: | The proposed model learns the relation completely or not at all on large datasets. |
The Curious Case of Absolute Position Embeddings (2022.findings-emnlp)
Copied to clipboard
| Challenge: | In natural language, it is not absolute position that matters, but relative position . et al., 2017) language models incorporate positional encodings that encode absolute (linear) word order. |
| Approach: | They find that Transformer language models encode word order using positional information . they also find that models that use absolute position embeddings over-rely on positional data . |
| Outcome: | The results raise questions about the efficacy of APEs to model the relativity of position information. |
Domain Regeneration: How well do LLMs match syntactic properties of text domains? (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent improvements in large language models have improved their ability to approximate distributions . authors find that LLMs can suffer from model collapse due to domain considerations based on pretraining . |
| Approach: | They use open source LLMs to regenerate permissively licensed English text from Wikipedia and news text. |
| Outcome: | The proposed model can faithfully match the human-generated distributions in a semantically-controlled setting. |
Information-Theoretic Probing for Linguistic Structure (2020.acl-main)
Copied to clipboard
| Challenge: | Neural networks are the backbone of modern stateof-the-art natural language processing systems. |
| Approach: | They propose an information-theoretic operationalization of probing as estimating mutual information that contradicts received wisdom . they evaluate on a set of ten typologically diverse languages often underrepresented in NLP research—plus English—totalling eleven languages. |
| Outcome: | The proposed model outperforms existing models on ten typologically diverse languages and English on 11 languages. |
XNLI: Evaluating Cross-lingual Sentence Representations (D18-1)
Copied to clipboard
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, Veselin Stoyanov
| Challenge: | State-of-the-art natural language processing systems rely on annotated data to learn competent models. |
| Approach: | They extend the development and test sets of the Multi-Genre Natural Language Inference Corpus to 14 languages, including Swahili and Urdu. |
| Outcome: | The proposed evaluation set extends the development and test sets of the Multi-Genre Natural Language Inference Corpus (MultiNLI) to 14 languages including low-resource languages such as Swahili and Urdu. |
Intrinsic Probing through Dimension Selection (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing research on probing for linguistic structure in word embeddings has focused on intrinsic probing, but what these representations encode about linguistic structures remains unclear. |
| Approach: | They propose a framework that allows us to determine whether linguistic information in word embeddings is dispersed or focal. |
| Outcome: | The proposed framework allows us to determine whether linguistic information in word embeddings is dispersed or focal. |
Arbiters of Ambivalence: Challenges of using LLMs in No-Consensus tasks (2025.findings-acl)
Copied to clipboard
| Challenge: | LLMs are increasingly being used to replace humans in "aligning" LLM training . studies question this trend, but have found they can be more effective in ambivalent scenarios where humans disagree . |
| Approach: | They develop a “no-consensus” benchmark by curating examples that encompass a variety of a priori ambivalent scenarios. |
| Outcome: | The proposed benchmarks show that LLMs can provide nuanced assessments when generating open-ended answers, but tend to take a stance on no-consensus topics when employed as judges or debaters. |
Perturbation Augmentation for Fairer NLP (2022.emnlp-main)
Copied to clipboard
| Challenge: | Unwanted and often harmful social biases are becoming more salient in NLP research. |
| Approach: | They propose to train a neural perturbation model that rewrites demographic references in text to make them more fair. |
| Outcome: | The proposed model outperforms heuristic alternatives on a large dataset of human annotated text perturbations. |
Sometimes We Want Ungrammatical Translations (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in Neural Machine Translation (NMT) systems focus on improving translation quality and improving robustness to perturbations. |
| Approach: | They propose a way to quantify faithfulness to the original text by focusing on word-order perturbations. |
| Outcome: | The proposed method aims to measure faithfulness and robustness in word-order perturbations without deleting or injecting tokens. |
UnNatural Language Inference (2021.acl-long)
Copied to clipboard
| Challenge: | Recent studies show that pre-trained NLU models understand human-like syntax . however, these models are word order invariant, causing them to assign gold labels to permutations . |
| Approach: | They propose to measure the severity of this issue by examining the properties of particular permutations that lead models to be word order invariant. |
| Outcome: | The proposed model is word order invariant, but it's not human-like syntax. |
Investigating Failures of Automatic Translation
in the Case of Unambiguous Gender (2022.acl-long)
Copied to clipboard
| Challenge: | Existing models are unable to make basic deductions regarding how to correctly inflect nouns with grammatical gender. |
| Approach: | They propose to evaluate NMT models' ability to translate gender morphology correctly in unambiguous contexts across syntactically diverse sentences. |
| Outcome: | The proposed model was unable to translate gender morphology correctly in unambiguous contexts across syntactically diverse sentences. |
On the Role of Speech Data in Reducing Toxicity Detection Bias (2025.naacl-long)
Copied to clipboard
Samuel Bell, Mariano Coria Meglioli, Megan Richards, Eduardo Sánchez, Christophe Ropers, Skyler Wang, Adina Williams, Levent Sagun, Marta R. Costa-jussà
| Challenge: | Text toxicity detection systems produce disproportionate rates of false positives on demographic groups . toxicity classification systems often misinterpret benign group mentions as toxic . |
| Approach: | They use group annotations to compare text-based and speech-based toxicity detection systems. |
| Outcome: | The results show that access to speech data supports reduced bias against group mentions . the authors recommend improving classifiers, rather than transcription pipelines if possible . |
ROBBIE: Robust Bias Evaluation of Large Generative Language Models (2023.emnlp-main)
Copied to clipboard
David Esiobu, Xiaoqing Tan, Saghar Hosseini, Megan Ung, Yuchen Zhang, Jude Fernandes, Jane Dwivedi-Yu, Eleonora Presani, Adina Williams, Eric Smith
| Challenge: | generative large language models (LLMs) are becoming more performant and prevalent . we need tools to measure and improve their fairness, authors say . |
| Approach: | They propose to compare 6 different prompt-based bias and toxicity metrics across 12 demographic axes and 5 families of generative large language models. |
| Outcome: | The proposed model can be tested on more datasets to better characterize and mitigate biases . the study compared 6 prompt-based bias and toxicity metrics across 12 demographic axes and 5 families of generative large language models. |
Quantifying the Semantic Core of Gender Systems (D19-1)
Copied to clipboard
| Challenge: | a large number of languages employ grammatical gender on the lexeme, but is it truly arbitrary? a recent study shows that the relationship between grammamatical gender and lexical semantics is opaque. |
| Approach: | They propose a method to correlating inanimate nouns' gender with lexical semantics . they find that the gender systems of 18 languages exhibit a significant correlation with a definition . |
| Outcome: | a new study shows that the gender assignments of 18 languages are arbitrary . the authors show that the correlation between gender and semantics is significant . |