Papers by Suchin Gururangan
Annotation Artifacts in Natural Language Inference Data (N18-2)
Copied to clipboard
| Challenge: | Large-scale datasets for natural language inference are created by crowdsourcing annotations . authors show that success of natural language models to date has been overestimated . |
| Approach: | They propose a method for crowdsourcing annotations to generate 3 new sentences based on a sentence (premise) they show that a simple text categorization model can correctly classify the hypothesis alone in about 67% of SNLI and 53% of MultiNLI . |
| Outcome: | The proposed model can classify the hypothesis alone in 67% of SNLI and 53% of MultiNLI datasets. |
Time Waits for No One! Analysis and Challenges of Temporal Misalignment (2022.naacl-main)
Copied to clipboard
| Challenge: | a pretrained model is optionally adapted through domain-specific pretraining, followed by task-specific finetuning. |
| Approach: | They establish a suite of eight tasks across different domains to quantify the effects of temporal misalignment in modern NLP systems. |
| Outcome: | The proposed tasks are based on eight domains and periods of time spanning five years or more and show that they have stronger effects than previous studies. |
Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selection (2022.emnlp-main)
Copied to clipboard
Suchin Gururangan, Dallas Card, Sarah Dreier, Emily Gade, Leroy Wang, Zeyu Wang, Luke Zettlemoyer, Noah A. Smith
| Challenge: | Language models rely on massive web crawls for diverse text data, but are rife with undesirable content. |
| Approach: | They analyze newspaper articles written by students from across the country to determine whose language is preferred by a quality filter. |
| Outcome: | The results show that newspapers from wealthier, educated, and urban zones are more likely to be classified as high quality. |
Detoxifying Language Models Risks Marginalizing Minority Voices (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing detoxification techniques have been proposed to mitigate toxic LM generations . e.g., detoxification makes LMs more brittle to distribution shift, especially on language used by marginalized groups . |
| Approach: | They propose to use detoxification techniques to reduce toxic LM generations without affecting perplexity or generation quality on nontoxic inputs. |
| Outcome: | The proposed methods hurt equity on language used by marginalized groups, the authors show . they show that detoxification makes LMs more brittle to distribution shift, they say . |
DEMix Layers: Disentangling Domains for Modular Language Modeling (2022.naacl-main)
Copied to clipboard
| Challenge: | Extensive experiments with autoregressive transformer LMs show that DEMix layers reduce test-time perplexity and increase training efficiency. |
| Approach: | They introduce a new domain expert mixture layer that enables conditioning a language model on the domain of the input text. |
| Outcome: | Experiments with 1.3B LMs show that DEMix layers reduce test-time perplexity, increase training efficiency, and enable rapid adaptation. |
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained neural language models (LMs) are prone to generating toxic language . we investigate the extent to which pretrained LMs can be prompted to generate toxic language. |
| Approach: | They propose to use a dataset to test the extent to which pretrained LMs can generate toxic language . they use toxicity-based text generation algorithms to generate toxic text from natural prompts . |
| Outcome: | The proposed model can degenerate into toxic text even from seemingly innocuous prompts. |
Variational Pretraining for Semi-supervised Text Classification (P19-1)
Copied to clipboard
| Challenge: | Empirically, we show the relative strength of VAMPIRE against computationally expensive contextual embeddings and other popular semi-supervised baselines under low resource settings. |
| Approach: | They propose a lightweight framework for effective text classification when data and computing resources are limited. |
| Outcome: | The proposed framework is compared with expensive contextual embeddings and semi-supervised baselines under low resource settings. |
Expected Validation Performance and Estimation of a Random Variable’s Maximum (2021.findings-emnlp)
Copied to clipboard
| Challenge: | In this paper, we analyze three statistical estimators for expected validation performance . Often researchers only report the performance of the best-found model during a hyperparameter search . |
| Approach: | They analyze three estimators for expected validation performance to compare models . they find that the estimator with the smallest variance has the largest bias . |
| Outcome: | The proposed model has the highest variance and the estimator with the smallest variance has the largest bias. |
Nearest Neighbor Zero-Shot Inference (2022.emnlp-main)
Copied to clipboard
| Challenge: | Using non-parametric memory for retrieval-augmented language models yields significant performance boosts over strong zeroshot baselines. |
| Approach: | They propose a retrieval-augmented language model with fuzzy verbalizers that expands the verbalizes that define different end-task class labels. |
| Outcome: | The proposed model outperforms non-retrieval-augmented language models on perplexity-based evaluations but gains transfer marginally . the main challenge is to achieve coverage of the verbalizer tokens that define the different end-task class labels. |
Show Your Work: Improved Reporting of Experimental Results (D19-1)
Copied to clipboard
| Challenge: | Current practice is to train multiple instantiations of each, choose the best model of each type, and compare their performance on held-out test data. |
| Approach: | They propose to measure expected validation accuracy as a function of computation budget . authors find comparisons where authors would have reached different conclusions if they had used more computation . |
| Outcome: | The proposed method shows that test-set performance scores alone are insufficient for drawing accurate conclusions about which model performs best. |
Self-Generated Critiques Boost Reward Modeling for Language Models (2025.naacl-long)
Copied to clipboard
Yue Yu, Zhengxing Chen, Aston Zhang, Liang Tan, Chenguang Zhu, Richard Yuanzhe Pang, Yundi Qian, Xuewei Wang, Suchin Gururangan, Chao Zhang, Melanie Kambadur, Dhruv Mahajan, Rui Hou
| Challenge: | Existing reward models produce scalar scores and struggle to incorporate critiques in a natural language format. |
| Approach: | They propose a framework that predicts critiques and rewards using self-generated critiques without extra supervision. |
| Outcome: | The proposed framework improves reward modeling accuracy by 3.7%-7.3% compared to standard reward models and LLM judges. |
Breaking the Curse of Multilinguality with Cross-lingual Expert Language Models (2024.emnlp-main)
Copied to clipboard
Terra Blevins, Tomasz Limisiewicz, Suchin Gururangan, Margaret Li, Hila Gonen, Noah Smith, Luke Zettlemoyer
| Challenge: | Multilingual language models often underperform monolingual ones due to inter-language competition for model parameters. |
| Approach: | They propose Cross-lingual Expert Language Models (X-ELM) which mitigates inter-language competition by independently training language models on subsets of the multilingual corpus. |
| Outcome: | The proposed model outperforms jointly trained multilingual models across all 16 considered languages and transfer the gains to downstream tasks. |
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks (2020.acl-main)
Copied to clipboard
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, Noah A. Smith
| Challenge: | Language models prerained on text from a wide variety of sources form the foundation of today’s NLP. |
| Approach: | They propose to tailor a pretrained model to the domain of a target task by using domain-adaptive pretraining in-domain. |
| Outcome: | The proposed model can be tailored to the domain of a target task and perform well under both high- and low-resource settings. |
BTS: Harmonizing Specialized Experts into a Generalist LLM (2025.emnlp-main)
Copied to clipboard
Qizhen Zhang, Prajjwal Bhargava, Chloe Bi, Chris X. Cai, Jakob Nicolaus Foerster, Jeremy Fu, Punit Singh Koura, Ruan Silva, Sheng Shen, Emily Dinan, Suchin Gururangan, Mike Lewis
| Challenge: | Branch-Train-Stitch (BTS) is an efficient and flexible training algorithm for combining independently trained large language model (LLM) experts into a single, capable generalist model. |
| Approach: | They propose an efficient and flexible training algorithm for combining large language model (LLM) experts into a single, capable generalist model using lightweight stitch layers. |
| Outcome: | The proposed model can generalize to new domains despite being frozen . it yields the best generalist performance on a variety of downstream tasks, retaining the specialized capabilities of each of the experts. |
Time is Encoded in the Weights of Finetuned Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Time vectors are used to customize language models to new time periods. |
| Approach: | They propose a tool to customize language models to new time periods by using time vectors . they show that time is encoded in the weight space of finetuned models . |
| Outcome: | The proposed tool improves performance on text from a time period without training. |
All That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text (2021.acl-long)
Copied to clipboard
| Challenge: | evaluators distinguish between human- and machine-authored text in three domains without training . evals' accuracy improved up to 55%, but it did not significantly improve across the three domain. |
| Approach: | They examine the role untrained human evaluations play in NLG evaluation and propose ways to improve their evaluations. |
| Outcome: | The evaluators distinguished between human- and machine-authored text at random chance level without training, but their accuracy did not improve across the three domains. |
AboutMe: Using Self-Descriptions in Webpages to Document the Effects of English Pretraining Data Filters (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models' (LLMs) abilities are drawn from their pretraining data. however, decisions around what data is retained or removed during this initial stage are under-scrutinized. |
| Approach: | They ground web text, a popular pretraining data source, to its social and geographic contexts. |
| Outcome: | The results show that some quality classifiers act like topical domain filters, and langID overlook English content from some regions of the world. |
M2D2: A Massively Multi-Domain Language Modeling Dataset (2022.emnlp-main)
Copied to clipboard
| Challenge: | M2D2 consists of 8.5B tokens and spans 145 domains extracted from Wikipedia and Semantic Scholar. |
| Approach: | They propose to organize 145 domains into 22 groups and use ontologies from Wikipedia and ArXiv to study domain adaptation in language models. |
| Outcome: | The proposed model enables the study of domain adaptation in language models (LMs) it shows that small amounts of fine-grained data can lead to larger in-domain performance gains than weakly relevant data. |