Papers by Christopher Manning
Backpack Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | We train a 170Mparameter Backpack language model on OpenWebText, matching the loss of a 6Bparameter Transformer. |
| Approach: | They propose a neural architecture that learns multiple non-contextual sense vectors for each word in a vocabulary and represents a word as a context-dependent, non-negative linear combination of sense vector. |
| Outcome: | The proposed model outperforms a GPT-2's word embeddings on lexical similarity evaluations and can be used to perform controllable text generation and debiasing. |
Grokking of Hierarchical Structure in Vanilla Transformers (2023.acl-short)
Copied to clipboard
| Challenge: | a recent study has shown that neural sequence models like transformers can generalize hierarchically when training for extended periods. |
| Approach: | They show that transformers can learn to generalize hierarchically after long training periods . they call this phenomenon structural grokking, which exhibits inverted U-shaped scaling in model depth . |
| Outcome: | The proposed model generalizes better than both very deep and very shallow models on multiple datasets. |
Enhancing Self-Consistency and Performance of Pre-Trained Language Models through Natural Language Inference (2022.emnlp-main)
Copied to clipboard
Eric Mitchell, Joseph Noh, Siyan Li, Will Armstrong, Ananth Agarwal, Patrick Liu, Chelsea Finn, Christopher Manning
| Challenge: | Existing frameworks for boosting consistency and accuracy of pre-trained NLP models without fine-tuning or re-training are lacking. |
| Approach: | They propose a framework for boosting the consistency and accuracy of pre-trained NLP models using pre-trainer natural language inference models without fine-tuning or re-training. |
| Outcome: | The proposed framework boosts consistency and accuracy of pre-trained NLP models using pre-train natural language inference models without fine-tuning or re-training. |
pyvene: A Library for Understanding and Improving PyTorch Models via Interventions (2024.naacl-demo)
Copied to clipboard
Zhengxuan Wu, Atticus Geiger, Aryaman Arora, Jing Huang, Zheng Wang, Noah Goodman, Christopher Manning, Christopher Potts
| Challenge: | Existing libraries are often project-based, but pyvene provides a unified and extensible framework for performing interventions on neural models and sharing the intervened upon models with others. |
| Approach: | They propose an open-source Python library that supports customizable interventions on a range of different PyTorch modules. |
| Outcome: | The proposed framework provides a unified and extensible framework for performing interventions on neural models and sharing the intervened upon models with others. |
You Only Need One Model for Open-domain Question Answering (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent approaches to Open-domain Question Answering use external knowledge bases, but have separate parameters and are weakly-coupled during training. |
| Approach: | They propose to use a single question answering model trained end-to-end to retrieve external knowledge and rerank passages with a separate reranked model. |
| Outcome: | The proposed model outperforms the previous state-of-the-art model by 1.0 and 0.7 exact match scores on the Natural Questions and TriviaQA open datasets. |
Do “English” Named Entity Recognizers Work Well on Global Englishes? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Most of English named entity recognition datasets contain American or British English data . multiple problems may occur in low-resource English contexts, such as confusion of named entities with regionspecific meanings . |
| Approach: | They build a newswire dataset to analyze NER model performance on low-resource English variants . they find that models trained on the CoNLL or OntoNotes datasets experienced significant performance drops . |
| Outcome: | The results show that models trained on the CoNLL or OntoNotes datasets experienced significant performance drops. |
MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for retraining from scratch are limited and only work on the recall of edited facts. |
| Approach: | They propose a benchmark method that allows users to ask multi-hop questions to assess whether edited models correctly answer questions where the answer should change as an entailed consequence of edited facts. |
| Outcome: | The proposed method outperforms existing models and scales well with LLMs (up to 175B) it is based on a memory-based approach that stores all edited facts externally while prompting the language model iteratively to generate answers consistent with the edited facts. |
Meta-Learning Online Adaptation of Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models encode impressively broad world knowledge in their parameters, but the knowledge in static models becomes stale. |
| Approach: | They propose to use a small, autoregressive model to reweight the language modeling loss for each token during online fine-tuning to maximize the out-of-date model’s ability to answer questions about a document. |
| Outcome: | The proposed approach improves the model's ability to answer questions on streams of thousands of documents compared with fine-tuning and baseline heuristics. |
Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback (2023.emnlp-main)
Copied to clipboard
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, Christopher Manning
| Challenge: | Recent studies have shown that unsupervised pre-training produces large language models whose conditional probabilities are remarkably well-calibrated. |
| Approach: | They propose to use verbalized confidences to extract confidence from large language models with reinforcement learning from human feedback to improve their accuracy. |
| Outcome: | The proposed methods reduce the expected calibration error by 50% for RLHF-LMs such as ChatGPT, GPT-4, and Claude. |
On Measuring the Intrinsic Few-Shot Hardness of Datasets (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent work has shown that few-shot learning is successful for pre-trained models, but there is no concrete understanding of when and why it is successful. |
| Approach: | They propose a simple metric that estimates few-shot hardness for a given dataset . they propose metric which exploits feature-space invariances between training and test samples . |
| Outcome: | The proposed metric better accounts for few-shot hardness compared to existing notions and is 8-100x faster to compute. |
Detecting Label Errors by Using Pre-Trained Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for label error detection focus on label errors in training data. |
| Approach: | They propose a method for introducing realistic, human-originated label noise into existing crowdsourced datasets such as SNLI and TweetNLP. |
| Outcome: | The proposed method outperforms existing methods for detecting label errors in natural language datasets. |
Answering Open-Domain Questions of Varying Reasoning Steps from Text (2021.emnlp-main)
Copied to clipboard
| Challenge: | a new benchmark is developed to answer open-domain questions from text . the system uses a single multi-task transformer model to perform all the necessary subtasks . |
| Approach: | They develop a unified system to answer directly from open-domain questions . they use a single multi-task transformer model to perform all the necessary subtasks . |
| Outcome: | The proposed system can answer open-domain questions on any text collection without prior knowledge of reasoning complexity. |
DReCa: A General Task Augmentation Strategy for Few-Shot Natural Language Inference (2021.naacl-main)
Copied to clipboard
| Challenge: | Meta-learning has not yet succeeded in NLP due to the lack of a well-defined task distribution . meta-learners tend to overfit their adaptation mechanism and datasets are heterogeneous . |
| Approach: | They propose a method for decomposing datasets into Reasoning Categories to form additional high quality tasks. |
| Outcome: | The proposed method improves the accuracy of meta-learners by 1.5-4% across four few-shot NLI problems. |
Mini But Mighty: Efficient Multilingual Pretraining with Linguistically-Informed Data Selection (2023.findings-eacl)
Copied to clipboard
| Challenge: | AfriBERTa shows that training transformer models from scratch on 1GB of data from many unrelated African languages outperforms massively multilingual models on downstream NLP tasks. |
| Approach: | They propose that training on smaller amounts of data but from related languages could match the performance of models trained on large, unrelated data. |
| Outcome: | The proposed model outperforms models trained on large, unrelated datasets on downstream NLP tasks. |
Truncation Sampling as Language Model Desmoothing (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Long samples of text from neural language models can be of poor quality. |
| Approach: | They propose to think of a neural language model as a mixture of k and a true distribution that avoids infinite perplexity. |
| Outcome: | The proposed methods generate more plausible long documents according to humans and break out of repetition. |
Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answering (2021.acl-long)
Copied to clipboard
| Challenge: | Currently, language-equipped vision systems such as VizWiz, TapTapSee, BeMyEyes, and CamFind are actively being deployed across a broad spectrum of users. |
| Approach: | They propose to identify collective outliers in active learning methods that are hard and often impossible for models to learn . they also propose to use visual inputs to identify these outlier examples as examples assigned low model confidence and prediction variability during training. |
| Outcome: | The proposed methods outperform random selection on visual question answering tasks. |
PragmatiCQA: A Dataset for Pragmatic Question Answering in Conversations (2023.findings-acl)
Copied to clipboard
| Challenge: | Mars? - PragmatiCQA |
| Approach: | Mars? - The Paper . |
| Outcome: | The proposed dataset features 6873 QA pairs that explores pragmatic reasoning in conversations over a diverse set of topics. |
Predicting Narratives of Climate Obstruction in Social Media Advertising (2024.findings-acl)
Copied to clipboard
| Challenge: | Social media advertising allows entities to construct narratives that align with their commercial interests and sway public perception. |
| Approach: | They propose to classify climate-related narratives into seven categories based on existing definitions and data. |
| Outcome: | The proposed method outperforms other methods and can reduce human annotation costs. |
Fixing Model Bugs with Natural Language Patches (2022.emnlp-main)
Copied to clipboard
| Challenge: | a growing body of research focused on using language to give instructions, supervision and even inductive biases to models instead of relying exclusively on labeled examples. |
| Approach: | They explore natural language patches that provide corrective feedback at the right level of abstraction. |
| Outcome: | The proposed model improves accuracy on real data by 1–4 accuracy points on different slices of a sentiment analysis dataset and F1 by 7 points on a relation extraction dataset. |
ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Contract review is a time-consuming procedure that costs companies millions of dollars each year . linguistic characteristics of contracts, such as negations by exceptions, contribute to the difficulty of this task . |
| Approach: | They propose a document-level natural language inference (NLI) task for contracts . they annotate and release the largest corpus to date consisting of 607 annotated contracts a linguistically rich system is proposed . |
| Outcome: | The proposed system is based on a contract review task that includes 607 annotated contracts. |
JamPatoisNLI: A Jamaican Patois Natural Language Inference Dataset (2022.findings-emnlp)
Copied to clipboard
| Challenge: | JamPatoisNLI provides the first dataset for natural language inference in a creole language, Jamaican Patois. |
| Approach: | They propose to use JamPatoisNLI to provide a dataset for natural language inference in a creole language, Jamaican Patois. |
| Outcome: | The proposed dataset shows that the language's relatedness to English significantly boosts the effectiveness of cross-lingual transfer even for the higher order task. |
Pushdown Layers: Encoding Recursive Structure in Transformer Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pushdown Layers model recursive state via stack tape that tracks estimated depths of tokens in incremental parsing . pushdown layers are drop-in replacement for standard self-attention . recursion is a key component of many aspects of intelligent behavior, authors say . |
| Approach: | They propose a self-attention layer that models recursive state via a stack tape . Pushdown Layers is a drop-in replacement for standard self- attention . |
| Outcome: | The proposed self-attention layer improves on parse tasks with a recursive-state model . it can model recursion using a stack tape that tracks estimated depths of tokens . |
Conditional probing: measuring usable information beyond a baseline (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for probing representations are limited to predicting part-of-speech . current methods cannot detect when a representation is predictive of just aspects of part- of-seech not explainable by the word identity. |
| Approach: | They propose to condition on the information in a baseline representation to test whether it is predictive of part-of-speech. |
| Outcome: | The proposed method is based on a theory of usable information called V-information and conditions on the information in the baseline. |
Human-like informative conversations: Better acknowledgements using conditional mutual information (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing chatbots generate responses that are non-specific w.r.t. one of the contexts, typically the conversational history. |
| Approach: | They propose to build a dialogue agent that can weave new factual content into conversations as naturally as humans. |
| Outcome: | The proposed method trades off pmi for pcmi_h and is preferred by humans for overall quality over the Max-PMI baseline 60% of the time. |
Statistical Uncertainty in Word Embeddings: GloVe-V (2024.emnlp-main)
Copied to clipboard
| Challenge: | Static word embeddings are ubiquitous in computational social science applications . however, assessing the statistical uncertainty in downstream conclusions remains challenging . |
| Approach: | They propose a method to obtain approximate, easy-to-use, and scalable reconstruction error variance estimates for one of the most widely used word embedding models. |
| Outcome: | The proposed method enables hypothesis testing in key word embedding tasks. |
LawInstruct: A Resource for Studying Language Model Adaptation to the Legal Domain (2025.findings-naacl)
Copied to clipboard
Joel Niklaus, Lucia Zheng, Arya D. McCarthy, Christopher Hahn, Brian M Rosen, Peter Henderson, Daniel E. Ho, Garrett Honke, Percy Liang, Christopher D Manning
| Challenge: | In general, instruction tuning is important for direct user interaction, but the legal domain is underrepresented in typical instruction datasets. |
| Approach: | They aggregate 58 annotated legal datasets and write instructions for each to create LawInstruct. |
| Outcome: | The proposed model improves on LegalBench across all model sizes, but no drop in MMLU. |