Papers by Jesse Dodge
Stubborn Lexical Bias in Data and Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent work has focused on spurious correlations between features and labels in training data . but, we find strong evidence of corresponding bias in the trained models . |
| Approach: | They propose a method to reduce spurious correlations in training data by reweighting it using a large pool of extracted features. |
| Outcome: | The proposed method reduces spurious correlations in training data, but still finds strong evidence of bias in trained models. |
Towards Reproducible Machine Learning Research in Natural Language Processing (2022.acl-tutorials)
Copied to clipboard
Ana Lucic, Maurits Bleeker, Samarth Bhargav, Jessica Forde, Koustuv Sinha, Jesse Dodge, Sasha Luccioni, Robert Stojnic
| Challenge: | a tutorial on reproducibility in ML addresses the problem of research results that are not reproducible. |
| Approach: | They propose a tutorial to ensure reproducible research in ML with an emphasis on computational linguistics and NLP. |
| Outcome: | The proposed tutorial focuses on computational linguistics and NLP . it provides a framework for using reproducibility as a teaching tool in university-level computer science programs. |
Efficient Hierarchical Domain Adaptation for Pretrained Language Models (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to allow domain adaptation to diverse domains are expensive and require continuing training in-domain. |
| Approach: | They propose a method to permit domain adaptation to many diverse domains using a computationally efficient adapter approach. |
| Outcome: | The proposed method allows domain adaptation to many diverse domains while avoiding negative interference between unrelated domains. |
The Right Tool for the Job: Matching Model and Instance Complexities (2020.acl-main)
Copied to clipboard
| Challenge: | a large increase in the size of NLP models can increase production costs and reduce adoption on real-time devices. |
| Approach: | They propose a modification to contextual representation fine-tuning which allows for an early exit from neural network calculations for simple instances and late exit for hard instances. |
| Outcome: | The proposed method produces models which are up to five times faster than the state of the art while preserving their accuracy. |
Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus (2021.emnlp-main)
Copied to clipboard
Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, Matt Gardner
| Challenge: | Large text corpora are often introduced with minimal documentation . documenting collection process, composition, intended uses, and other are key for structured, task-specific datasets. |
| Approach: | They propose to document a dataset created by applying filters to a single snapshot of Common Crawl. |
| Outcome: | The proposed dataset shows that blocklist filtering removes text from minority individuals and patents. |
OLMoTrace: Tracing Language Model Outputs Back to Trillions of Training Tokens (2025.acl-demo)
Copied to clipboard
Jiacheng Liu, Taylor Blanton, Yanai Elazar, Sewon Min, Yen-Sung Chen, Arnavi Chheda-Kothary, Huy Tran, Byron Bischoff, Eric Marsh, Michael Schmitz, Cassidy Trier, Aaron Sarnat, Jenna James, Jon Borchardt, Bailey Kuehl, Evie Yu-Yen Cheng, Karen Farley, Taira Anderson, David Albright, Carissa Schoenick, Luca Soldaini, Dirk Groeneveld, Rock Yuren Pang, Pang Wei Koh, Noah A. Smith, Sophie Lebrecht, Yejin Choi, Hannaneh Hajishirzi, Ali Farhadi, Jesse Dodge
| Challenge: | tracing language models' outputs back to training data is a problem because they are trained on text corpora with trillions of tokens . existing methods for tracers have not been scaled to work within this multi-trillion-token setting . |
| Approach: | They propose a system that traces language models' outputs verbatim back to training data . OLMOTRACE retrieves documents from the model's training data that contain exact matches . |
| Outcome: | The proposed system can find verbatim matches between LM output and training data . it can be used to explore fact checking, hallucination, and creativity of language models . |
Efficient Methods for Natural Language Processing: A Survey (2023.tacl-1)
Copied to clipboard
Marcos Treviso, Ji-Ung Lee, Tianchu Ji, Betty van Aken, Qingqing Cao, Manuel R. Ciosici, Michael Hassid, Kenneth Heafield, Sara Hooker, Colin Raffel, Pedro H. Martins, André F. T. Martins, Jessica Zosa Forde, Peter Milder, Edwin Simpson, Noam Slonim, Jesse Dodge, Emma Strubell, Niranjan Balasubramanian, Leon Derczynski, Iryna Gurevych, Roy Schwartz
| Challenge: | Recent work in natural language processing (NLP) has yielded appealing results from scaling model parameters and training data, but using only scale to improve performance means resource consumption also grows. |
| Approach: | They propose to use data, time, storage, or energy to improve model performance. |
| Outcome: | The proposed methods and findings provide guidance for conducting NLP under limited resources and point towards promising research directions for developing more efficient methods. |
OLMES: A Standard for Language Model Evaluations (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing models claim to perform better on tasks measuring model capabilities, but there is no standard setup for reproducible evaluations. |
| Approach: | They propose a document that is documented and practical for reproducible LLM evaluations and includes recommendations from existing literature and new experiments. |
| Outcome: | The proposed standard identifies and reviews the varying factors in evaluation practices adopted by the community, such as prompt formatting, choice of in-context examples, probability normalizations, and task formulation. |
Expected Validation Performance and Estimation of a Random Variable’s Maximum (2021.findings-emnlp)
Copied to clipboard
| Challenge: | In this paper, we analyze three statistical estimators for expected validation performance . Often researchers only report the performance of the best-found model during a hyperparameter search . |
| Approach: | They analyze three estimators for expected validation performance to compare models . they find that the estimator with the smallest variance has the largest bias . |
| Outcome: | The proposed model has the highest variance and the estimator with the smallest variance has the largest bias. |
RNN Architecture Learning with Sparse Regularization (D19-1)
Copied to clipboard
| Challenge: | State-of-the-art NLP models require hundreds of millions and even billions of parameters to perform, which can lead to memory usage and increased runtime. |
| Approach: | They propose a structure learning method that uses group lasso to learn sparse, parameter-efficient NLP models by pruning more than 90% of the weights of rational RNNs. |
| Outcome: | The proposed method learns sparse, parameter-efficient models without sacrificing performance relative to parameter-rich baselines. |
Language Models Hallucinate, but May Excel at Fact Verification (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have produced non-factual outputs . however, current LLMs suffer from the hallucination issue . |
| Approach: | They propose to use instruction-tuned LLMs to generate factual outputs . they find that FLAN-T5-11B performs best as a fact verifier . |
| Outcome: | The proposed method outperforms more capable LLMs like GPT3.5 and ChatGPT in the human evaluation. |
Scalable Data Ablation Approximations for Language Models through Modular Training and Merging (2024.emnlp-main)
Copied to clipboard
Clara Na, Ian Magnusson, Ananya Harsh Jha, Tom Sherborne, Emma Strubell, Jesse Dodge, Pradeep Dasigi
| Challenge: | Training data compositions for Large Language Models (LLMs) can significantly affect their downstream performance. |
| Approach: | They propose a method which trains individual models on subsets of a training corpus and reuses them across evaluations of combinations of subset. |
| Outcome: | The proposed method improves training efficiency by scaling only linearly with respect to new data. |
Show Your Work: Improved Reporting of Experimental Results (D19-1)
Copied to clipboard
| Challenge: | Current practice is to train multiple instantiations of each, choose the best model of each type, and compare their performance on held-out test data. |
| Approach: | They propose to measure expected validation accuracy as a function of computation budget . authors find comparisons where authors would have reached different conclusions if they had used more computation . |
| Outcome: | The proposed method shows that test-set performance scores alone are insufficient for drawing accurate conclusions about which model performs best. |
Words as Gatekeepers: Measuring Discipline-specific Terms and Meanings in Scholarly Publications (2023.findings-acl)
Copied to clipboard
| Challenge: | Specialized terminology, or specialized language, can facilitate efficient in-group communication within fields but hinder understanding for out-groups. |
| Approach: | They propose an interpretable approach to measure scholarly jargon from text using word sense induction to identify words that are widespread but overloaded with different meanings across fields. |
| Outcome: | The proposed method is validated by word sense induction and shows that it provides a complementary, yet unique view of jargon alongside word types. |
AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Models (2023.findings-eacl)
Copied to clipboard
| Challenge: | Pretrained language models often need to specialize to specific domains. |
| Approach: | They propose an approach that performs weight-space averaging of adapters trained on different domains. |
| Outcome: | The proposed approach improves performance to new domains without extra training. |
Reproducibility in NLP: What Have We Learned from the Checklist? (2023.findings-acl)
Copied to clipboard
| Challenge: | Scientific progress in NLP rests on the reproducibility of researchers’ claims. |
| Approach: | They examine 10,405 anonymous responses to the NLP Reproducibility Checklist . they find evidence of an increase in reporting of information after the Checklist's introduction . |
| Outcome: | The authors find that 44% of submissions that gather new data are 5% less likely to be accepted than those that did not. |
Competency Problems: On Finding and Removing Artifacts in Language Data (2021.emnlp-main)
Copied to clipboard
Matt Gardner, William Merrill, Jesse Dodge, Matthew Peters, Alexis Ross, Sameer Singh, Noah A. Smith
| Challenge: | Recent work in NLP has documented dataset artifacts, bias, and spurious correlations . how to tell which features have spurious instead of legitimate correlations is typically left unspecified . |
| Approach: | They propose a class of competency problems to formalize this notion into a classification . they show that realistic datasets will increasingly deviate from competency problems . |
| Outcome: | The proposed model can be used to show that models are inappropriately affected by these less extreme biases. |
Merge to Learn: Efficiently Adding Skills to Language Models with Model Merging (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Adapting general-purpose language models to new skills is currently expensive . Adaptation to new skill sets requires repeated training or models forget older skills . |
| Approach: | They propose a parallel-train-then-merge procedure that adds new skills to preexisting models in isolation and later merges with the general model. |
| Outcome: | The proposed method is cheaper than retraining models on updated datasets . it improves model compliance with safe prompts while preserving model's ability to refuse dangerous or harmful prompts. |
AboutMe: Using Self-Descriptions in Webpages to Document the Effects of English Pretraining Data Filters (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models' (LLMs) abilities are drawn from their pretraining data. however, decisions around what data is retained or removed during this initial stage are under-scrutinized. |
| Approach: | They ground web text, a popular pretraining data source, to its social and geographic contexts. |
| Outcome: | The results show that some quality classifiers act like topical domain filters, and langID overlook English content from some regions of the world. |