Papers by Christopher Potts

Mittens: an Extension of GloVe for Learning Domain-Specialized Representations (N18-2)

Challenge:	a multilingual spelling correction model is needed to meet the tight latency requirements of multilingual NLP . a monolingual teacher model is trained for each language/locale, and individual models are distilled into a single student model .
Approach:	They propose a multilingual approach to spelling correction using multi-teacher distillation . they train a monolingual teacher model for each language and distill them into a single model .
Outcome:	The proposed model can meet the tight latency requirements of deployed services.

Copied to clipboard

Nicholas Dingwall, Christopher Potts

Representation Learning

Challenge:	We show that the resulting representations can lead to faster learning and better results on a variety of tasks.
Approach:	They propose a simple extension of the GloVe representation learning model that starts with general-purpose representations and updates them based on specialized data sets.
Outcome:	The proposed model synthesizes general-purpose representations with specialized data while remaining faithful to the original space.

BioDEX: Large-Scale Biomedical Adverse Drug Event Extraction for Real-World Pharmacovigilance (2023.findings-emnlp)

Copied to clipboard

Karel D’Oosterlinck, François Remy, Johannes Deleu, Thomas Demeester, Chris Develder, Klim Zaporojets, Aneiss Ghodsi, Simon Ellershaw, Jack Collins, Christopher Potts

Information Extraction & Text Mining

Challenge:	pharmacovigilance (PV) is a tool for analyzing adverse drug events from biomedical literature . pharmacologists use natural language processing to extract core information from papers .
Approach:	They propose a resource for biomedical adverse drug event eXtraction using natural language processing.
Outcome:	The proposed model achieves 59.1% F1 (validation) and estimates human performance to be 72.0% F1 . the proposed model could be used to improve drug safety monitoring, also called pharmacovigilance, in the future.

Causal Interventions Reveal Shared Structure Across English Filler–Gap Constructions (2025.emnlp-main)

Copied to clipboard

Sasha Boguraev, Christopher Potts, Kyle Mahowald

Updating CLIP to Prefer Descriptions Over Captions (2024.emnlp-main)

Challenge:	Language Models (LMs) have emerged as powerful sources of evidence for linguists seeking to develop theories of syntax.
Approach:	They propose to use causal interpretability methods to characterize abstract mechanisms that LMs learn to use by transferring a wh-filler-gap structure into a gap-less c++ class.
Outcome:	The proposed methods can characterize the abstract mechanisms that LMs learn to use, and challenge claims that they can be learned only with strong innate priors.

Copied to clipboard

Amir Zur, Elisa Kreiss, Karel D’Oosterlinck, Christopher Potts, Atticus Geiger

pyvene: A Library for Understanding and Improving PyTorch Models via Interventions (2024.naacl-demo)

Challenge:	Current metrics for imagetext similarity tend to be insensitive to the text's purpose.
Approach:	They propose to use a model that assigns higher scores to descriptions than captions . they use parameter efficient fine-tuning and a loss objective to shed light on the distinction .
Outcome:	The proposed model correlates with the judgements of blind and low-vision people while preserving transfer capabilities and sheds light on the caption–description distinction.

Copied to clipboard

Zhengxuan Wu, Atticus Geiger, Aryaman Arora, Jing Huang, Zheng Wang, Noah Goodman, Christopher Manning, Christopher Potts

Explainability & Interpretability in NLP

Challenge:	Existing libraries are often project-based, but pyvene provides a unified and extensible framework for performing interventions on neural models and sharing the intervened upon models with others.
Approach:	They propose an open-source Python library that supports customizable interventions on a range of different PyTorch modules.
Outcome:	The proposed framework provides a unified and extensible framework for performing interventions on neural models and sharing the intervened upon models with others.

RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representations (2024.acl-long)

Copied to clipboard

Jing Huang, Zhengxuan Wu, Christopher Potts, Mor Geva, Atticus Geiger

TalkDown: A Corpus for Condescension Detection in Context (D19-1)

Challenge:	Existing methods to disentangle individual neurons from multiple high-level concepts are not yet benchmarked.
Approach:	They propose a method of Multi-task Distributed Alignment Search that allows to find distributed representations satisfying multiple causal criteria.
Outcome:	The proposed method achieves state-of-the-art on the target language model with Llama2-7B .

Copied to clipboard

Zijian Wang, Christopher Potts

Challenge:	condescending language use can bring dialogues to an end and disrupt healthy communities.
Approach:	They propose a model that uses a language-only model to model condescending linguistic acts in context.
Outcome:	a new model of condescending language use improves performance and motivates techniques . the model can estimate condescension rates in various online communities and relate these differences to community norms .

Relevance-guided Supervision for OpenQA with ColBERT (2021.tacl-1)

Copied to clipboard

Omar Khattab, Christopher Potts, Matei Zaharia

Information Retrieval

Challenge:	Recent work has focused on learning to retrieve passages for open-domain question answering . if notions of relevance are not tailored to questions, the MRC model will not reliably see the best passages .
Approach:	They propose a retrieval model that uses coarse-grained vector representations of questions and passages to adapt it to OpenQA.
Outcome:	The proposed system improves OpenQA retrieval on Natural Questions, SQuAD, and TriviaQA.

MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions (2023.emnlp-main)

Copied to clipboard

Zexuan Zhong, Zhengxuan Wu, Christopher Manning, Christopher Potts, Danqi Chen

Retrofitting Distributional Embeddings to Knowledge Graphs with Functional Relations (C18-1)

Challenge:	Existing methods for retraining from scratch are limited and only work on the recall of edited facts.
Approach:	They propose a benchmark method that allows users to ask multi-hop questions to assess whether edited models correctly answer questions where the answer should change as an entailed consequence of edited facts.
Outcome:	The proposed method outperforms existing models and scales well with LLMs (up to 175B) it is based on a memory-based approach that stores all edited facts externally while prompting the language model iteratively to generate answers consistent with the edited facts.

Copied to clipboard

Ben Lengerich, Andrew Maas, Christopher Potts

Lexical Semantics with Large Language Models: A Case Study of English “break” (2023.findings-eacl)

Challenge:	Existing methods for retrofitting knowledge graph embeddings assume connected entities have similar embeddments, but these assumptions are not true for large knowledge graphs.
Approach:	They propose to retrofit distributional and relational data to a knowledge graph structure . they propose to explicitly model pairwise relations to overcome these limitations .
Outcome:	The proposed framework outperforms existing retrofitting methods on complex knowledge graphs and loses no accuracy on simpler graphs.

Copied to clipboard

Erika Petersen, Christopher Potts

Pragmatic Issue-Sensitive Image Captioning (2020.findings-emnlp)

Challenge:	Large neural language models (LLMs) can be powerful tools for research in lexical semantics.
Approach:	They argue that large neural language models can be powerful tools for research in lexical semantics by capturing known sense distinctions and identifying informative new sense combinations.
Outcome:	The proposed models capture many of the sense distinctions found in the English verb break and can be used to identify informative new sense combinations for further analysis.

Copied to clipboard

Allen Nie, Reuben Cohn-Gordon, Christopher Potts

AmazonQAC: A Large-Scale, Naturalistic Query Autocomplete Dataset (2024.emnlp-industry)

Challenge:	Issue-Sensitive Image Captioning (ISIC) is a new approach to image captioning . high-quality captions are shaped by the communicative goal of identifying the target image .
Approach:	They propose to use image partitions to control image caption generation to produce descriptive captions.
Outcome:	The proposed model can be extended to include image partitions and image partitioning.

Copied to clipboard

Dante Everaert, Rohit Patki, Tianqi Zheng, Christopher Potts

Dynabench: Rethinking Benchmarking in NLP (2021.naacl-main)

Challenge:	Existing systems that provide a graphical representation of QAC are limited in their ability to provide real-time data.
Approach:	They introduce a new QAC dataset sourced from Amazon Search logs . they assess Prefix Trees, semantic retrieval, and Large Language Models with and without finetuning .
Outcome:	The proposed system can predict search terms based on user-typed prefixes . the proposed system achieves only half of what is theoretically possible on the test data .

Copied to clipboard

Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, Adina Williams

Challenge:	Dynabench is an open-source platform for dynamic dataset creation and model benchmarking.
Approach:	They propose an open-source platform for dynamic dataset creation and model benchmarking.
Outcome:	The proposed platform can be used to create models that fail on simple challenges and falter in real-world scenarios.

Representing Social Media Users for Sarcasm Detection (D18-1)

Copied to clipboard

Y. Alex Kolchinski, Christopher Potts

Sentiment Analysis

Challenge:	Existing annotated corpus of Reddit comments is limited by available annotation methods.
Approach:	They propose a Bayesian approach that directly represents authors’ propensities to be sarcastic and a dense embedding approach that can learn interactions between the author and the text.
Outcome:	The proposed approach performs better in homogeneous contexts, whereas the dense embeddings prove valuable in more diverse contexts.

Systematicity in GPT-3’s Interpretation of Novel English Noun Compounds (2022.findings-emnlp)

Copied to clipboard

Siyan Li, Riley Carlson, Christopher Potts

Data and Representation for Turkish Natural Language Inference (2020.emnlp-main)

Challenge:	e.g., stew skillet, swamp squash) are not fully compositional, but highly predictable based on whether the modifier and head refer to artifacts or natural kinds.
Approach:	They propose to compare the interpretations of novel English noun compounds with the large language model GPT-3, which is governed by interpretive principles.
Outcome:	The results show that the large language model GPT-3 reasoning only about specific lexical items is consistent with the Levin et al.'s theory.

Copied to clipboard

Emrah Budur, Rıza Özçelik, Tunga Gungor, Christopher Potts

Multilinguality

Challenge:	Large annotated datasets in NLP are overwhelmingly in English . obtaining new annotation resources for each task in each language would be prohibitively expensive .
Approach:	They propose to use machine translation to translate large annotated datasets into Turkish . they find that in-language embeddings are essential and morphological parsing can be avoided .
Outcome:	The proposed model trains on human-translated evaluation sets.

Recursive Routing Networks: Learning to Compose Modules for Language Understanding (N19-1)

Copied to clipboard

Ignacio Cases, Clemens Rosenbaum, Matthew Riemer, Atticus Geiger, Tim Klinger, Alex Tamkin, Olivia Li, Sandhini Agarwal, Joshua D. Greene, Dan Jurafsky, Christopher Potts, Lauri Karttunen

Challenge:	Recursive Routing Networks are modular, adaptable models that learn effectively in diverse environments.
Approach:	They propose to apply Recursive Routing Networks (RRNs) to natural language understanding by integrating them into existing architectures and recurrent network hidden layers.
Outcome:	The proposed model optimizes the parameters of the functions and the meta-learner decision-making component for routing inputs through those functions.

Inducing Character-level Structure in Subword-based Language Models with Type-level Interchange Intervention Training (2023.findings-acl)

Copied to clipboard

Jing Huang, Zhengxuan Wu, Kyle Mahowald, Christopher Potts

Responsible & Trustworthy NLP

Challenge:	Language models operating on subword units are challenging for character-level manipulations, authors say . authors develop a framework to learn robust character representations inside subword-based models .
Approach:	They propose a causal intervention framework to learn robust character representations inside subword-based language models.
Outcome:	The proposed model outperforms character-level models on more complex tasks . it improves robustness on unseen token sequences and leads to human-interpretable representations of characters.

Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs (2024.emnlp-main)

Copied to clipboard

Krista Opsahl-Ong, Michael Ryan, Josh Purtell, David Broman, Christopher Potts, Matei Zaharia, Omar Khattab

DynaSent: A Dynamic Benchmark for Sentiment Analysis (2021.acl-long)

Challenge:	Language Model Programs (LMs) require crafting prompts that are jointly effective for all modules.
Approach:	They propose a novel algorithm for optimizing language model (LM) prompts for all modules by using program- and data-aware techniques and stochastic mini-batch evaluation functions.
Outcome:	The proposed algorithm outperforms baseline optimizers on five of seven diverse LM programs by as high as 13% accuracy.

Copied to clipboard

Christopher Potts, Zhengxuan Wu, Atticus Geiger, Douwe Kiela

Sentiment Analysis

Challenge:	Sentiment analysis is an early success story for NLP, in both a technical and an industrial sense.
Approach:	They propose to combine naturally occurring sentences with sentences created using the open-source Dynabench Platform, which facilities human-and-model-in-the-loop dataset creation.
Outcome:	The proposed model is more coherent than comparable models and motivates training models from scratch over successive fine-tuning.

Moving Beyond Downstream Task Accuracy for Information Retrieval Benchmarking (2023.findings-acl)

Copied to clipboard

Keshav Santhanam, Jon Saad-Falcon, Martin Franz, Omar Khattab, Avi Sil, Radu Florian, Md Arafat Sultan, Salim Roukos, Matei Zaharia, Christopher Potts

Green & Sustainable NLP

Challenge:	Neural information retrieval (IR) systems have progressed rapidly in recent years . many IR benchmarks focus on downstream task accuracy, concealing costs incurred .
Approach:	They propose to include efficiency considerations on IR benchmarks to help drive progress . eral et al. propose to incorporate query latency and cost budgets into evaluation .
Outcome:	a new study shows that the best IR system varies according to how efficiency considerations are chosen and weighed . the proposed benchmarks would allow for more thorough exploration of possible system designs .

False Friends Are Not Foes: Investigating Vocabulary Overlap in Multilingual Language Models (2025.findings-emnlp)

Copied to clipboard

Julie Kallini, Dan Jurafsky, Christopher Potts, Martijn Bartelds

Multilinguality

Challenge:	Prior work has shown that token overlap facilitates cross-lingual transfer or introduces interference between languages?
Approach:	They devised a controlled experiment where they train bilingual autoregressive models on multiple language pairs under systematically varied vocabulary overlap settings.
Outcome:	The proposed model outperforms models with disjointed vocabularies on XNLI and XQuAD and shows that token overlap is beneficial for multilingual tokenizers.

Causal Distillation for Language Models (2022.naacl-main)

Copied to clipboard

Zhengxuan Wu, Atticus Geiger, Joshua Rozner, Elisa Kreiss, Hanson Lu, Thomas Icard, Christopher Potts, Noah Goodman

Demystifying Verbatim Memorization in Large Language Models (2024.emnlp-main)

Challenge:	Distillation efforts have led to language models that are more compact and efficient without serious drops in performance.
Approach:	They propose to augment distillation with a third objective that encourages the student model to imitate the causal dynamics of the teacher through a distillation interchange intervention training objective (DIITO).
Outcome:	The proposed method lowers perplexity on the WikiText-103M corpus and improves on the GLUE benchmark, SQuAD, and CoNLL-2003.

Copied to clipboard

Jing Huang, Diyi Yang, Christopher Potts

Context Matters for Image Descriptions for Accessibility: Challenges for Referenceless Evaluation Metrics (2022.emnlp-main)

Challenge:	Existing studies have shown that Large Language Models (LLMs) memorize long sequences verbatim, with serious copyright and privacy implications.
Approach:	They develop a framework to study verbatim memorization in a controlled setting by continuing pre-training from Pythia checkpoints with injected sequences.
Outcome:	The proposed framework creates a control model M () and a treatment model M with injected sequences.

Copied to clipboard

Elisa Kreiss, Cynthia Bennett, Shayan Hooshmand, Eric Zelikman, Meredith Ringel Morris, Christopher Potts

Visual Data in NLP

Challenge:	Existing referenceless metrics do not take context into account, whereas contextual information is highly valued by BLV users.
Approach:	They propose a contextual version of the referenceless metric CLIPScore which addresses the disconnect to the BLV data.
Outcome:	The proposed evaluation metrics are based on a proof-of-concept with blind and low vision (BLV) participants.

UDAPDR: Unsupervised Domain Adaptation via LLM Prompting and Distillation of Rerankers (2023.emnlp-main)

Copied to clipboard

Jon Saad-Falcon, Omar Khattab, Keshav Santhanam, Radu Florian, Martin Franz, Salim Roukos, Avirup Sil, Md Sultan, Christopher Potts

Responsible & Trustworthy NLP

Challenge:	Existing methods for information retrieval tasks require large labeled datasets for fine-tuning, but they can experience significant drops in accuracy due to distribution shifts from the training to the target domain.
Approach:	They propose a method for using large language models to generate large numbers of synthetic queries cheaply using an expensive LLM.
Outcome:	The proposed method boosts zero-shot accuracy in long-tail domains and achieves substantially lower latency than standard reranking methods.

ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction (2022.naacl-main)

Copied to clipboard

Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, Matei Zaharia

ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems (2024.naacl-long)

Challenge:	Neural information retrieval (IR) methods encode queries and documents into single vectors, but late interaction models produce multi-vector representations at the granularity of each token.
Approach:	They propose a retrieval method that couples an aggressive residual compression mechanism with a denoised supervision strategy to improve the quality and space footprint of late interaction.
Outcome:	The proposed retriever improves quality and space footprint of late interaction models while reducing space footprint by 6–10x.

Copied to clipboard

Jon Saad-Falcon, Omar Khattab, Christopher Potts, Matei Zaharia

Information Retrieval

Challenge:	Evaluating retrieval-augmented generation systems relies on hand annotations for input queries, passages to retrieve, and responses to generate.
Approach:	They propose an automated evaluation framework for retrieval-augmented generation (RAG) ARES fine tunes lightweight LLM judges on synthetically generated queries and answers .
Outcome:	The proposed framework evaluates RAG systems using only human annotations . it can be used to improve system understanding and create targeted solutions .

Pragmatically Informative Image Captioning with Character-Level Inference (N18-2)

Copied to clipboard

Reuben Cohn-Gordon, Noah Goodman, Christopher Potts

Posing Fair Generalization Tasks for Natural Language Inference (D19-1)

Challenge:	a neural image captioner and a Rational Speech Acts (RSA) model are pragmatically informative . previous attempts to combine RSA with neural image-captioning require an inference which normalizes over the entire set of possible utterances.
Approach:	They propose a neural image captioner with a Rational Speech Acts model to make it pragmatically informative.
Outcome:	The proposed system outperforms a non-pragmatic baseline and word-level RSA captioner on a word-based model.

Copied to clipboard

Atticus Geiger, Ignacio Cases, Lauri Karttunen, Christopher Potts

Textual Inference

Challenge:	Existing evaluation methods for deep learning semantics rely on naturalistic corpora, but they often fail to support the kind of generalization we are asking for.
Approach:	They define and motivate a formal notion of fairness for evaluations of deep learning models for semantics . they then apply it to natural language inference by constructing challenging but provably fair artificial datasets based on the results .
Outcome:	The proposed evaluations show that standard neural models fail to generalize in the required ways and even these models do not solve the task perfectly.

Retrieval Augmented Spelling Correction for E-Commerce Applications (2024.emnlp-industry)

Copied to clipboard

Xuan Guo, Rohit Patki, Dante Everaert, Christopher Potts

Information Retrieval

Challenge:	e-commerce spelling correction services face a challenge with new brand names . we propose a new approach that uses a fine-tuned retrieval algorithm to correct for brand names.
Approach:	They propose a method that uses product names to be incorporated into a large language model to do contextual spelling correction.
Outcome:	The proposed approach improves performance with only minor latency increases . the proposed approach is more efficient than a stand-alone LLM .

Generating Bilingual Pragmatic Color References (N18-1)

Copied to clipboard

Will Monroe, Jennifer Hu, Andrew Jong, Christopher Potts

Multilinguality

Challenge:	Contextual influences on language often exhibit substantial cross-lingual regularities, but are obscured by semantic and syntactic differences.
Approach:	They propose a model that captures language-specific syntax and semantics while also exhibiting responsiveness to contextual difficulty in Chinese and English.
Outcome:	The proposed model can identify synonyms between the two languages, even with no exposure to parallel data.

ScoNe: Benchmarking Negation Reasoning in Language Models With Fine-Tuning and In-Context Learning (2023.acl-short)

Copied to clipboard

Jingyuan S. She, Christopher Potts, Samuel R. Bowman, Atticus Geiger

I am a Strange Dataset: Metalinguistic Tests for Language Models (2024.acl-long)

Challenge:	Negation is a ubiquitous but complex linguistic phenomenon that poses a significant challenge for NLP systems.
Approach:	They propose a benchmark that measures how well models handle natural language negation . they extend ScoNe-NLI to embed negation reasoning in short narratives .
Outcome:	The proposed model can reason about negation, but struggles to do so on NLI examples outside of its core pretraining regime.

Copied to clipboard

Tristan Thrush, Jared Moore, Miguel Monares, Christopher Potts, Douwe Kiela

Detecting Contradictory COVID-19 Drug Efficacy Claims from Biomedical Literature (2023.acl-short)

Challenge:	Existing datasets for metalinguistic self-reference are limited by the number of subtasks.
Approach:	They propose a dataset that aims to address metalinguistic self-reference in large language models.
Outcome:	The proposed dataset is hand-crafted by experts and validated by non-expert annotators.

Copied to clipboard

Daniel Sosa, Malavika Suresh, Christopher Potts, Russ Altman

Reasoning

Challenge:	During times of pandemic, treatment options are limited, and developing new drug treatments is infeasible in the short-term.
Approach:	They propose to use a natural language inference problem to automatically identify contradictory claims about COVID-19 drug efficacy.
Outcome:	The proposed models help domain experts distill and assess evidence concerning remdisivir and hydroxychloroquine.

Fine-Tuning and Prompt Optimization: Two Great Steps that Work Better Together (2024.emnlp-main)

Copied to clipboard

Dilara Soylu, Christopher Potts, Omar Khattab

Concadia: Towards Image-Based Text Generation with a Purpose (2022.emnlp-main)

Challenge:	Recent work shows the potential of building more powerful Natural Language Processing systems by composing multiple skills of LMs into pipelines.
Approach:	They propose to combine weight and prompt optimization strategies to optimize a modular LM pipeline.
Outcome:	The proposed strategies outperform optimizing weights and prompts alone by 60% and 6% on average across LMs and tasks.

Copied to clipboard

Elisa Kreiss, Fei Fang, Noah Goodman, Christopher Potts