Papers by Luke Zettlemoyer

FaVIQ: FAct Verification from Information-seeking Questions (2022.acl-long)

Challenge:	Existing methods to condition models on a concise rationale are less accurate than models that can use the entire context.
Approach:	They propose a method to optimize a bound on the Information Bottleneck objective to extract concise rationales from a binary mask and an end-task predictor that uses only the residual sentences.
Outcome:	The proposed model outperforms existing norm-minimization techniques in task performance and agreement with human rationales in the ERASER benchmark.

Copied to clipboard

Jungsoo Park, Sewon Min, Jaewoo Kang, Luke Zettlemoyer, Hannaneh Hajishirzi

Crowdsourcing Question-Answer Meaning Representations (N18-2)

Challenge:	Existing fact verification datasets with crowdsourced claims introduce subtle biases that are difficult to control for.
Approach:	They construct a large-scale fact verification dataset with ambiguous questions . they use a corpus of 188k claims to construct false and true claims .
Outcome:	The proposed dataset outperforms models trained on the dataset FEVER or in-domain data by up to 17% absolute.

Copied to clipboard

Julian Michael, Gabriel Stanovsky, Luheng He, Ido Dagan, Luke Zettlemoyer

Representation Learning

Challenge:	Existing datasets for predicate-argument relationships are lacking highly skilled and trained annotators.
Approach:	They propose a crowdsourcing scheme to generate question-answer pairs that represent predicate-argument relationships in sentences as a set of question-announcer pairs.
Outcome:	The proposed model covers the vast majority of predicate-argument relationships in existing datasets along with many previously under-resourced ones, including implicit arguments and relations.

Translate to Disambiguate: Zero-shot Multilingual Word Sense Disambiguation with Pretrained Language Models (2024.eacl-long)

Copied to clipboard

Haoqiang Kang, Terra Blevins, Luke Zettlemoyer

Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Models (2022.emnlp-main)

Challenge:	Pretrained language models learn cross-lingual knowledge and perform well on diverse tasks when finetuned.
Approach:	They propose a zero-shot prompting approach that captures cross-lingual word sense with a contextual prompt.
Outcome:	The proposed approach outperforms baselines on recall in many evaluation languages without additional training or finetuning.

Copied to clipboard

Terra Blevins, Hila Gonen, Luke Zettlemoyer

Questions Are All You Need to Train a Dense Passage Retriever (2023.tacl-1)

Challenge:	Existing studies on multilingual models have focused on their cross-lingual transfer behavior . a recent study examined multilingual model learning from the multilingual pretraining signal .
Approach:	They analyze checkpoints during multilingual pretraining to identify when models acquire in-language and cross-lingual abilities.
Outcome:	The proposed model achieves high in-language performance early on, with lower-level linguistic skills acquired before more complex ones.

Copied to clipboard

Devendra Singh Sachan, Mike Lewis, Dani Yogatama, Luke Zettlemoyer, Joelle Pineau, Manzil Zaheer

Information Retrieval

Challenge:	Existing methods for dense retrieval require large supervised datasets with custom hard-negative mining and denoising of positive examples.
Approach:	They propose a new corpus-level autoencoding approach for training dense retrieval models that does not require labeled training data.
Outcome:	The proposed method matches or surpasses strong supervised performance levels on multiple QA benchmarks with no labeled training data or task-specific losses.

Low-Resource Domain Adaptation for Compositional Task-Oriented Semantic Parsing (2020.emnlp-main)

Copied to clipboard

Xilun Chen, Asish Ghoshal, Yashar Mehdad, Luke Zettlemoyer, Sonal Gupta

Deep Contextualized Word Representations (N18-1)

Challenge:	Recent advances in deep learning have enabled several approaches to successfully parse more complex queries, but these models require a large amount of annotated training data to parser on new domains (e.g. reminder, music).
Approach:	They propose a method that adapts task-oriented semantic parsers to low-resource domains and outperforms a supervised neural model at a 10-fold data reduction.
Outcome:	The proposed method outperforms baseline methods on a newly collected multi-domain task-oriented semantic parsing dataset (TOPv2) .

Copied to clipboard

Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, Luke Zettlemoyer

QANom: Question-Answer driven SRL for Nominalizations (2020.coling-main)

Challenge:	a new type of deep contextualized word representation is proposed for language understanding problems . word vectors are learned functions of the internal states of a deep bidirectional language model .
Approach:	They propose a new type of deep contextualized word representation that models complex features of word use and how they vary across linguistic contexts.
Outcome:	The proposed representations improve the state of the art across six challenging NLP problems.

Copied to clipboard

Ayal Klein, Jonathan Mamou, Valentina Pyatkin, Daniela Stepanov, Hangfeng He, Dan Roth, Luke Zettlemoyer, Ido Dagan

Eunsol Choi, Omer Levy, Yejin Choi, Luke Zettlemoyer

Challenge:	Traditionally, SRL annotations focus on verbal predicates, but other types of predicate are frequent in natural language.
Approach:	They propose a semantic scheme for capturing predicate-argument relations for nominalizations, termed QANom, using crowdsourcing and QA-driven annotations.
Outcome:	The proposed scheme outperforms existing annotations and is useful for downstream tasks.

Ultra-Fine Entity Typing (P18-1)

Copied to clipboard

SpanBERT: Improving Pre-training by Representing and Predicting Spans (2020.tacl-1)

Challenge:	Experimental results show that a model that can predict ultra-fine types can be crowd-sourced . head words indicate the type of the noun phrases they appear in, and are important for context-sensitive tasks .
Approach:	They propose a task where sentences are given with an entity mention . they introduce a new type of distant supervision: head words, which indicate the type of noun phrases they appear in.
Outcome:	The proposed model can predict ultra-fine types at varying granularity and performs well on a fine-grained entity typing benchmark.

Copied to clipboard

Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, Omer Levy

MULTIGUARD: An Efficient Approach for AI Safety Moderation Across Languages and Modalities (2025.emnlp-main)

Challenge:	Pre-training methods like BERT mask individual words or subword units, but many tasks involve reasoning about relationships between two or more spans of text.
Approach:	They propose a pre-training method that masks contiguous random spans instead of random tokens to train the span boundary representations to predict the entire content of the masked span.
Outcome:	The proposed method outperforms BERT and its better-tuned baselines on span selection tasks and on coreference resolution tasks.

Copied to clipboard

Sahil Verma, Keegan Hines, Jeff Bilmes, Charlotte Siska, Luke Zettlemoyer, Hila Gonen, Chandan Singh

Syntactic Scaffolds for Semantic Structures (D18-1)

Challenge:	Existing approaches to detect harmful queries to large language models are fallible and vulnerable to attacks that exploit mismatched generalization of model capabilities.
Approach:	They propose an approach to detect harmful queries to large language models (LLMs) OMNIGUARD identifies internal representations of an LLM/MLLM that are aligned across languages or modalities and builds a language-agnostic or modality-adic classifier for detecting harmful prompts.
Outcome:	OMNIGUARD improves harmful prompt classification accuracy by 11.57% over the strongest baseline in a multilingual setting, by 20.44% for image-based prompts, and sets a new SOTA for audio-based ones.

Copied to clipboard

Swabha Swayamdipta, Sam Thomson, Kenton Lee, Luke Zettlemoyer, Chris Dyer, Noah A. Smith

Syntactic Text Processing

Challenge:	Syntactic scaffolds avoid expensive syntactical processing at runtime . many systems have used syntastic dependency or phrase-based parsers as preprocessing for semantic analysis.
Approach:	They propose a multitask learning approach that uses a syntactic treebank to integrate syntaktic information into semantic tasks.
Outcome:	The proposed method improves on PropBank semantics, frame semantics and coreference resolution tasks.

SimpleQuestions Nearly Solved: A New Upperbound and Baseline Approach (D18-1)

Copied to clipboard

Michael Petrochuk, Luke Zettlemoyer

Challenge:	ambiguity in the data bounds performance of the SimpleQuestions dataset, which is commonly used for factoid questions . ambiguities are a problem because many questions have more than one equally plausible interpretation .
Approach:	They propose a benchmark that can be solved by standard methods using the SimpleQuestions dataset . they propose ambiguity in the data bounds performance at 83.4% and a baseline that sets a new state-of-the-art performance level at 78.1% accuracy .
Outcome:	The SimpleQuestions dataset is one of the most commonly used benchmarks for studying factoids . the new benchmark is 78.1% accurate, and the upperbound is loose, the authors show .

Simple and Effective Retrieve-Edit-Rerank Text Generation (2020.acl-main)

Copied to clipboard

Nabil Hossain, Marjan Ghazvininejad, Luke Zettlemoyer

CREPE: Open-Domain Question Answering with False Presuppositions (2023.acl-long)

Challenge:	Using retrieve-and-edit methods, text generation methods can be improved by reranking outputs from training sets and learning models to produce the final output.
Approach:	They propose to extend retrieve-and-edit seq2seq methods with a simple post-generation ranking approach that retrieves multiple outputs and edits each independently to produce the final output.
Outcome:	The proposed approach outperforms existing methods on two machine translation datasets and shows room for improvement with better candidate output selection in future work.

Copied to clipboard

Xinyan Yu, Sewon Min, Luke Zettlemoyer, Hannaneh Hajishirzi

Muppet: Massive Multi-task Representations with Pre-Finetuning (2021.emnlp-main)

Challenge:	Existing question answering datasets assume all questions have well defined answers.
Approach:	They propose a QA dataset containing a distribution of false presuppositions . they find that 25% of questions contain false presumptions .
Outcome:	The proposed model finds that 25% of questions contain false presuppositions . the model can find presuffpositions moderately well, but struggle when predicting correctness .

Copied to clipboard

Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer, Sonal Gupta

Improving Factuality with Explicit Working Memory (2025.acl-long)

Challenge:	Recent work shows gains from pre-training and fine-tuning that are multi-task . but it can be difficult to know which intermediate tasks will best transfer .
Approach:	They propose a large-scale learning stage for pre-finetuning between pre-training and fine-tun.
Outcome:	The proposed model improves performance on pretrained discriminators and generation models on a wide range of tasks while improving sample efficiency during fine-tuning.

Copied to clipboard

Mingda Chen, Yang Li, Karthik Padthe, Rulin Shao, Alicia Yi Sun, Luke Zettlemoyer, Gargi Ghosh, Wen-tau Yih

Information Retrieval

Challenge:	Large language models can generate factually inaccurate content, a problem known as hallucination.
Approach:	They propose an approach that integrates a working memory that receives feedback from external resources.
Outcome:	The proposed method outperforms baselines on four fact-seeking datasets and increases the factuality metric by 2 to 6 points absolute.

Efficient Large Scale Language Modeling with Mixtures of Experts (2022.emnlp-main)

Copied to clipboard

Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, Giridharan Anantharaman, Xian Li, Shuohui Chen, Halil Akin, Mandeep Baines, Louis Martin, Xing Zhou, Punit Singh Koura, Brian O’Horo, Jeffrey Wang, Luke Zettlemoyer, Mona Diab, Zornitsa Kozareva, Veselin Stoyanov

Few-shot Learning with Multilingual Generative Language Models (2022.emnlp-main)

Challenge:	Mixture of Experts layers (MoEs) enable efficient scaling of language models . large autoregressive language models such as GPT-3 can be adapted to a wide range of tasks .
Approach:	They propose to use Mixture of Experts layers to enable efficient scaling of language models . they find that MoEs are substantially more compute efficient than dense models compared to MoE models - but only when they are more modestly trained .
Outcome:	The proposed model outperforms dense models in a wide range of tasks and domains.

Copied to clipboard

Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, Xian Li

Nonparametric Masked Language Modeling (2023.findings-acl)

Challenge:	Large-scale generative language models such as GPT-3 are competitive few-shot learners.
Approach:	They train multilingual generative language models on a corpus covering a diverse set of languages and study their few- and zero-shot learning capabilities.
Outcome:	The proposed model outperforms GPT-3 on 171 out of 182 directions with 32 training examples and surpasses the official supervised baseline in 45 directions.

Copied to clipboard

Sewon Min, Weijia Shi, Mike Lewis, Xilun Chen, Wen-tau Yih, Hannaneh Hajishirzi, Luke Zettlemoyer

Evaluating Gender Bias in Machine Translation (P19-1)

Challenge:	Existing language models (LMs) predict tokens with a softmax over a finite vocabulary, which can make it difficult to predict rare tokens or phrases.
Approach:	They introduce a nonparametric masked language model that replaces a softmax with a distribution over every phrase in a reference corpus and uses an in-batch approximation to train it.
Outcome:	The proposed model outperforms larger parametric models on 16 tasks including classification, fact probing and question answering.

Copied to clipboard

Gabriel Stanovsky, Noah A. Smith, Luke Zettlemoyer

Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters (2023.acl-long)

Challenge:	Using morphological analysis, we find that MT models exhibit gender-biased translation errors when training data encode stereotypes not relevant for the task.
Approach:	They propose an automatic gender bias evaluation method for eight target languages with grammatical gender based on morphological analysis.
Outcome:	The proposed method is based on two recent coreference resolution datasets composed of English sentences cast participants into non-stereotypical gender roles.

Copied to clipboard

Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, Huan Sun

Low-Resource NLP

Challenge:	Chain-of-Thought (CoT) prompting can dramatically improve the multi-step reasoning abilities of large language models (LLMs).
Approach:	They propose to use Chain-of-Thought (CoT) prompting to encourage the LLM to generate intermediate rationales for solving a problem by providing a series of reasoning steps in the demonstrations.
Outcome:	The proposed model can generate coherent lines of reasoning even with invalid demonstrations while still generating coherent lines during inference.

Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selection (2022.emnlp-main)

Copied to clipboard

Suchin Gururangan, Dallas Card, Sarah Dreier, Emily Gade, Leroy Wang, Zeyu Wang, Luke Zettlemoyer, Noah A. Smith

Prompting Contrastive Explanations for Commonsense Reasoning Tasks (2021.findings-acl)

Challenge:	Language models rely on massive web crawls for diverse text data, but are rife with undesirable content.
Approach:	They analyze newspaper articles written by students from across the country to determine whose language is preferred by a quality filter.
Outcome:	The results show that newspapers from wealthier, educated, and urban zones are more likely to be classified as high quality.

Copied to clipboard

Bhargavi Paranjape, Julian Michael, Marjan Ghazvininejad, Hannaneh Hajishirzi, Luke Zettlemoyer

Prompting Language Models for Linguistic Structure (2023.acl-long)

Challenge:	Large pretrained language models (PLMs) can achieve near-human performance on commonsense reasoning tasks, but provide little human-interpretable evidence of the underlying reasoning they use.
Approach:	They propose to use large pretrained language models to generate evidence for commonsense reasoning NLP tasks . they use models to contrast alternative explanations based on key attribute(s) required to justify the correct answer .
Outcome:	The proposed model improves performance on two commonsense reasoning benchmarks compared to previous non-contrastive alternatives.

Copied to clipboard

Terra Blevins, Hila Gonen, Luke Zettlemoyer

The Referential Reader: A Recurrent Entity Network for Anaphora Resolution (P19-1)

Challenge:	Existing prompting methods can test this hypothesis on autoregressive PLMs.
Approach:	They propose a structured prompting approach for linguistic structured prediction tasks that performs zero- and few-shot sequence tagging with autoregressive PLMs.
Outcome:	The proposed approach shows that the model can perform few-shot sequence tagging on part-of-speech taging, named entity recognition, and sentence chunking tasks.

Copied to clipboard

Fei Liu, Luke Zettlemoyer, Jacob Eisenstein

Multilingual Denoising Pre-training for Neural Machine Translation (2020.tacl-1)

Challenge:	Existing methods for storing and accessing entity mentions are expensive and implausible for human readers.
Approach:	They propose a method for storing and accessing entity mentions during online text processing.
Outcome:	The proposed model performs well on a dataset of pronoun-name anaphora.

Copied to clipboard

Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, Luke Zettlemoyer

MetaICL: Learning to Learn In Context (2022.naacl-main)

Challenge:	Existing approaches to pre-train models focus on only English corpora, but this is not common in machine translation.
Approach:	They propose a sequence-to-sequence denoising auto-encoder pre-trained on monolingual corpora . they show that it produces significant performance gains across MT tasks .
Outcome:	The proposed model can achieve significant performance gains across a wide variety of MT tasks.

Copied to clipboard

Sewon Min, Mike Lewis, Luke Zettlemoyer, Hannaneh Hajishirzi

DEMix Layers: Disentangling Domains for Modular Language Modeling (2022.naacl-main)

Challenge:	Large language models can do in-context learning by conditioning on a few training examples with no parameter updates or task-specific templates.
Approach:	They propose a meta-training framework where a pretrained language model is tuned to do in-context learning on a large set of training tasks.
Outcome:	The proposed framework outperforms baseline models on 142 NLP datasets and a range of target tasks with domain shifts.

Copied to clipboard

Suchin Gururangan, Mike Lewis, Ari Holtzman, Noah A. Smith, Luke Zettlemoyer

Learning to Model and Ignore Dataset Bias with Mixed Capacity Ensembles (2020.findings-emnlp)

Challenge:	Extensive experiments with autoregressive transformer LMs show that DEMix layers reduce test-time perplexity and increase training efficiency.
Approach:	They introduce a new domain expert mixture layer that enables conditioning a language model on the domain of the input text.
Outcome:	Experiments with 1.3B LMs show that DEMix layers reduce test-time perplexity, increase training efficiency, and enable rapid adaptation.

Copied to clipboard

Christopher Clark, Mark Yatskar, Luke Zettlemoyer

Visual Data in NLP

Challenge:	Recent work has shown that datasets contain incidental correlations created by idiosyncrasies in the data collection process.
Approach:	They propose a method that detects and ignores dataset-specific correlations by introducing a new method that makes them conditionally independent.
Outcome:	The proposed method detects and ignores these kinds of dataset-specific correlations, and does not require the bias to be known in advance.

Dissecting Contextual Word Embeddings: Architecture and Representation (D18-1)

Copied to clipboard

Matthew E. Peters, Mark Neumann, Luke Zettlemoyer, Wen-tau Yih

Unsupervised Cross-lingual Representation Learning at Scale (2020.acl-main)

Challenge:	Existing work on learning contextual representations has used LSTM-based biLMs, but there is no reason to believe this is effective.
Approach:	They propose to use pre-trained bidirectional language models to learn contextual word embeddings for four NLP tasks and to use them to study the effects of architecture on endtask accuracy.
Outcome:	The proposed models outperform word embeddings for four NLP tasks and all learn representations that vary with network depth.

Copied to clipboard

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, Veselin Stoyanov

Z-ICL: Zero-Shot In-Context Learning with Pseudo-Demonstrations (2023.acl-long)

Challenge:	Pretraining multilingual language models at scale leads to performance gains for cross-lingual transfer tasks.
Approach:	They present a transformer-based multilingual masked language model pre-trained on 100 languages . they show that pretraining multilingual models at scale leads to significant performance gains .
Outcome:	The proposed model outperforms multilingual BERT (mBERT) on cross-lingual benchmarks.

Copied to clipboard

Xinxi Lyu, Sewon Min, Iz Beltagy, Luke Zettlemoyer, Hannaneh Hajishirzi

Low-Resource NLP

Challenge:	Existing methods for zero-shot learning are based on in-context training, but performance drops when no demonstrations are available.
Approach:	They propose a new method that constructs pseudo-demonstrations for a given test input using a raw text corpus and applies techniques to reduce copying.
Outcome:	The proposed method outperforms previous zero-shot methods on nine classification datasets and is on par with in-context learning with labeled training data in the few-shot setting.

DESCGEN: A Distantly Supervised Datasetfor Generating Entity Descriptions (2021.acl-long)

Copied to clipboard

Weijia Shi, Mandar Joshi, Luke Zettlemoyer

Knowledge Representation

Challenge:	Short textual descriptions of entities provide summaries of their key attributes but generating entity descriptions can be challenging since information is scattered across multiple sources with varied content and style.
Approach:	They propose to generate an entity summary description from 37K entities from Wikipedia and Fandom, paired with nine evidence documents on average.
Outcome:	The proposed task is entity-centric, more abstractive, and covers a wide range of domains.

pair2vec: Compositional Word-Pair Embeddings for Cross-Sentence Inference (N19-1)

Copied to clipboard

Mandar Joshi, Eunsol Choi, Omer Levy, Daniel Weld, Luke Zettlemoyer

Representation Learning

Challenge:	Existing inference models that rely heavily on unsupervised single-word embeddings struggle to learn implied relationships between pairs of words.
Approach:	They propose to use word embeddings to learn and use background knowledge about implied relationships between words that are crucial for cross-sentence inference problems.
Outcome:	The proposed models gain 2.7% on the recently released SQuAD 2.0 and 1.3% on MultiNLI, and 8.8% on the adversarial SQu AD datasets.

VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding (2021.findings-acl)

Copied to clipboard

Hu Xu, Gargi Ghosh, Po-Yao Huang, Prahal Arora, Masoumeh Aminzadeh, Christoph Feichtenhofer, Florian Metze, Luke Zettlemoyer

On the Role of Bidirectionality in Language Model Pre-Training (2022.findings-emnlp)

Challenge:	Existing methods for multimodal video understanding are task-specific, limiting their use for retrieval-style end tasks.
Approach:	They propose a task-agnostic multimodal pre-training approach that can accept video or text input, or both, for a variety of end tasks.
Outcome:	The proposed approach outperforms existing methods on a wider range of tasks while maintaining separability.

Copied to clipboard

Mikel Artetxe, Jingfei Du, Naman Goyal, Luke Zettlemoyer, Veselin Stoyanov

Language Contamination Helps Explains the Cross-lingual Capabilities of English Pretrained Models (2022.emnlp-main)

Challenge:	Prior work on language model pre-training explored different architectures and learning objectives, but differences in data, hyperparameters and evaluation make a principled comparison difficult.
Approach:	They propose a framework that generalizes prior approaches to pre-training language models by focusing on bidirectionality and controlling each of them separately.
Outcome:	The proposed framework generalizes prior approaches including fully unidirectional models like GPT, fully bidirectional models and hybrid models like CM3 and prefix LM.

Copied to clipboard

Terra Blevins, Luke Zettlemoyer

Inducing Semantic Roles Without Syntax (2021.findings-acl)

Challenge:	a large number of pretraining corpora are not publicly available, and it is unclear how much foreign language data exists in monolingual models.
Approach:	They propose to use English pretraining corpora to analyze their language composition . they find that even when less than 1% of data is not English, it facilitates cross-lingual transfer .
Outcome:	The proposed model is not truly monolingual when pretrained at scale, the authors show . they show that even when less than 1% of data is not English, it facilitates cross-lingual transfer .

Copied to clipboard

Julian Michael, Luke Zettlemoyer

Syntactic Text Processing

Challenge:	Semantic roles are a key component of linguistic predicate-argument structure, but syntax can be difficult to define, annotate, and predict.
Approach:	They propose to use QA-SRL to automatically induce semantic roles from ontologies that use question-answer pairs to represent predicate-argument structure.
Outcome:	The proposed method outperforms existing models and a state-of-the-art model over gold syntax.

Trusting Your Evidence: Hallucinate Less with Context-aware Decoding (2024.naacl-short)

Copied to clipboard

Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, Wen-tau Yih

Span-based Hierarchical Semantic Parsing for Task-Oriented Dialog (D19-1)

Challenge:	Language models (LMs) often struggle to pay enough attention to the input context, and generate texts that are unfaithful or contain hallucinations.
Approach:	They propose a context-aware decoding technique that amplifies the difference between the output probabilities when a model is used with and without context.
Outcome:	The proposed model significantly improves faithfulness of different LM families including OPT, GPT, LLaMA, and FLAN-T5 for summarization tasks.

Copied to clipboard

Panupong Pasupat, Sonal Gupta, Karishma Mandyam, Rushin Shah, Mike Lewis, Luke Zettlemoyer

Compositional Questions Do Not Necessitate Multi-hop Reasoning (P19-1)

Challenge:	Existing semantic parsers score intents and slots as labels of nesting nodes, but decode a valid tree globally.
Approach:	They propose a span-based semantic parser for parsing compositional utterances into Task Oriented Parse (TOP) the parsers score labels of the tree nodes covering each token span independently, but decode a valid tree globally.
Outcome:	The proposed parser outperforms previous methods on the TOP dataset in accuracy and training speed.

Copied to clipboard

Sewon Min, Eric Wallace, Sameer Singh, Matt Gardner, Hannaneh Hajishirzi, Luke Zettlemoyer

Reasoning

Challenge:	a single-hop reasoning model can solve much more of the dataset than previously thought.
Approach:	They propose a single-hop BERT-based RC model that achieves 67 F1 . they propose an evaluation setting where humans are not shown all paragraphs .
Outcome:	The proposed model achieves 67 F1—comparable to state-of-the-art multi-hop models.

Prompt-free and Efficient Few-shot Learning with Language Models (2022.acl-long)

Copied to clipboard

Rabeeh Karimi Mahabadi, Luke Zettlemoyer, James Henderson, Lambert Mathias, Marzieh Saeidi, Veselin Stoyanov, Majid Yazdani

XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models (2023.emnlp-main)

Challenge:	Existing methods for few-shot fine-tuning of pretrained language models require carefully engineered prompts and verbalizers to convert inputs into a cloze-format that the PLM can score.
Approach:	They propose a method for few-shot fine-tuning of pretrained language models that uses task-specific adapters instead of manually engineered prompts and verbalizers.
Outcome:	The proposed method outperforms existing state-of-the-art methods on a wide range of few shot NLP tasks.

Copied to clipboard

Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Naman Goyal, Marjan Ghazvininejad, Luke Zettlemoyer, Madian Khabsa

FEWS: Large-Scale, Low-Shot Word Sense Disambiguation with the Dictionary (2021.eacl-main)

Challenge:	Large multilingual models rely on a single vocabulary shared across 100+ languages . this vocabulary bottleneck limits the representational capabilities of multilingual model XLM-R .
Approach:	They propose a new approach for scaling to large multilingual vocabularies by de-emphasizing token sharing between languages with little lexical overlap and assigning vocabulary capacity to achieve sufficient coverage for each individual language.
Outcome:	The proposed model outperforms XLM-R on all language tasks and is particularly effective on low-resource tasks.

Copied to clipboard

Terra Blevins, Mandar Joshi, Luke Zettlemoyer

Bilingual Lexicon Induction via Unsupervised Bitext Construction and Word Alignment (2021.acl-long)

Challenge:	Existing models for Word Sense Disambiguation struggle to disambiguate rare senses . current models struggle to learn senses with few training examples .
Approach:	They introduce a low-shot WSD dataset automatically extracted from example sentences in Wiktionary.
Outcome:	The proposed dataset outperforms baseline models on rare senses in existing datasets.

Copied to clipboard

Haoyue Shi, Luke Zettlemoyer, Sida I. Wang

JuICe: A Large Scale Distantly Supervised Dataset for Open Domain Context-based Code Generation (D19-1)

Challenge:	Existing methods for bilingual lexicon induction are linear and require simplifying assumptions.
Approach:	They propose methods that combine unsupervised bitext mining and unsupervised word alignment to produce higher quality lexicons.
Outcome:	The proposed method outperforms the state-of-the-art on the BUCC 2020 task by 14 F1 points . further analysis suggests they are comparable quality .

Copied to clipboard

Rajas Agashe, Srinivasan Iyer, Luke Zettlemoyer

Programming Languages in NLP

Challenge:	Interactive programming with interleaved code snippet cells and natural language markdown is gaining popularity in the form of Jupyter notebooks.
Approach:	They propose to train code generation models based on a corpus of 1.5 million examples with a curated test set of 3.7K instances based off online programming assignments.
Outcome:	The proposed model generates code cells based on the NL-Code history and human-curated data.

Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning (2021.acl-long)

Copied to clipboard

Armen Aghajanyan, Sonal Gupta, Luke Zettlemoyer

Does Liking Yellow Imply Driving a School Bus? Semantic Leakage in Language Models (2025.naacl-long)

Challenge:	Pre-trained language models can be fine-tuned to produce state-of-the-art results for a wide range of language understanding tasks.
Approach:	They propose to analyze fine-tuning through the lens of intrinsic dimension . they show that pre-trained models have a low intrinsic dimension reparameterization .
Outcome:	The proposed model can achieve 90% of the full parameter performance levels on MRPC with low data regime.

Copied to clipboard

Hila Gonen, Terra Blevins, Alisa Liu, Luke Zettlemoyer, Noah A. Smith

Contrastive Decoding: Open-ended Text Generation as Optimization (2023.acl-long)

Challenge:	Despite their wide adoption, the biases and unintended behaviors of language models remain poorly understood.
Approach:	They propose an evaluation setting to detect semantic leakage by humans and automatically . they also curate a diverse test suite for diagnosing this behavior in 13 flagship models .
Outcome:	The proposed evaluation setting detects semantic leakage by humans and automatically, and measures it in 13 flagship models.

Copied to clipboard

Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, Mike Lewis

Supervised Open Information Extraction (N18-1)

Challenge:	Using a language model, maximum probability is a poor decoding objective because it produces short and repetitive text.
Approach:	They propose a reliable decoding approach that optimizes a contrastive objective subject to a plausibility constraint.
Outcome:	The proposed approach outperforms four strong decoding algorithms in automatic and human evaluations across wikipedia, news and story domains.

Copied to clipboard

Gabriel Stanovsky, Julian Michael, Luke Zettlemoyer, Ido Dagan

Quantifying Adaptability in Pre-trained Language Models with 500 Tasks (2022.naacl-main)

Challenge:	Existing methods for Open Information Extraction (Open IE) use semisupervised approaches or rule-based algorithms.
Approach:	They propose a supervised approach to Open Information Extraction (Open IE) they build on recent deep Semantic Role Labeling models to extract Open IE tuples .
Outcome:	The proposed model outperforms state-of-the-art Open IE systems on benchmark datasets.

Copied to clipboard

Belinda Li, Jane Yu, Madian Khabsa, Luke Zettlemoyer, Alon Halevy, Jacob Andreas

CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation (2022.findings-emnlp)

Challenge:	a recent study examines the features and limits of LM adaptability to new tasks . many questions about the nature and limits remain unanswered .
Approach:	They evaluate adaptability to new tasks using a new benchmark, TaskBench500 . they find adaptation procedures differ dramatically in their ability to memorize small datasets .
Outcome:	The proposed benchmark compares 500 procedurally generated sequence modeling tasks to a new benchmark.

Copied to clipboard

Tanay Dixit, Bhargavi Paranjape, Hannaneh Hajishirzi, Luke Zettlemoyer

s1: Simple test-time scaling (2025.emnlp-main)

Challenge:	Prior work on counterfactual data augmentation only considered restricted classes of perturbations, limiting their effectiveness.
Approach:	They propose a retrieval-augmented framework for creating diverse counterfactual perturbations for CDA.
Outcome:	Experiments on natural language inference and sentiment analysis show that the proposed framework can be used to encourage diversity in manually authored perturbations.

Copied to clipboard

Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candes, Tatsunori Hashimoto

Reasoning

Challenge:	OpenAI’s o1 model showed this capability but did not publicly share its methodology, leading to many replication efforts.
Approach:	They curate a small dataset s1K with 1,000 reasoning questions based on three criteria we validate through ablations: difficulty, diversity, and quality.
Outcome:	The proposed model exceeds o1-preview on competition math questions by up to 27% (MATH and AIME24).

A Discrete Hard EM Approach for Weakly Supervised Question Answering (D19-1)

Copied to clipboard

Sewon Min, Danqi Chen, Hannaneh Hajishirzi, Luke Zettlemoyer

Improving Passage Retrieval with Zero-Shot Question Generation (2022.emnlp-main)

Challenge:	Existing work on question answering tasks only provide weak supervision for how the answer should be computed . weak supervision is attractive because it is relatively easy to gather, allowing for large datasets . but weak supervision complicates learning because there are many different spurious ways to derive the correct answer.
Approach:	They propose a method to convert question answering tasks into discrete latent variable learning problems with a precomputed set of possible solutions that contains one correct option.
Outcome:	The proposed approach outperforms previous methods on six QA tasks and achieves state-of-the-art on five of them.

Copied to clipboard

Devendra Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan, Wen-tau Yih, Joelle Pineau, Luke Zettlemoyer

Low-Resource NLP

Challenge:	Existing re-ranking methods for open-domain question answering are not domain- or task-specific.
Approach:	They propose a simple and effective re-ranking method for improving passage retrieval in open-domain question answering.
Outcome:	The proposed method outperforms strong supervised models on open-domain questions and triviaQA datasets on top-1000 passages.

Deep RNNs Encode Soft Hierarchical Syntax (P18-2)

Copied to clipboard

Terra Blevins, Omer Levy, Luke Zettlemoyer

Syntactic Text Processing

Challenge:	Existing studies show that syntactic information is useful for a wide variety of NLP tasks.
Approach:	They propose to use word-level representations to learn internal representations that capture soft hierarchical notions of syntax from highly varied supervision.
Outcome:	The proposed model encodes significant amounts of syntax even without explicit supervision.

Natural Language to Code Translation with Execution (2022.emnlp-main)

Copied to clipboard

Freda Shi, Daniel Fried, Marjan Ghazvininejad, Luke Zettlemoyer, Sida I. Wang

Don’t Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases (D19-1)

Challenge:	Generative code models do not explicitly incorporate program semantics during training, but they are able to generate correct solutions for many problems.
Approach:	They introduce execution result-based minimum Bayes risk decoding for program selection . they select output programs from a generated candidate set by marginalizing over implementations that share the same semantics .
Outcome:	The proposed model outperforms all other methods on natural language-to-code translation.

Copied to clipboard

Christopher Clark, Mark Yatskar, Luke Zettlemoyer

Robustness in NLP

Challenge:	Recent advances in neural models exploit dataset-specific patterns that do not generalize well to out-of-domain or adversarial settings.
Approach:	They propose to train a model to be more robust to domain shift if it has prior knowledge of dataset biases.
Outcome:	The proposed model can be more robust to domain shift if it has prior knowledge of dataset biases.

REPLUG: Retrieval-Augmented Black-Box Language Models (2024.naacl-long)

Copied to clipboard

Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Richard James, Mike Lewis, Luke Zettlemoyer, Wen-tau Yih

The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants (2024.acl-long)

Challenge:	Existing retrieval-augmented language models require access to internal representations to enhance performance.
Approach:	They introduce a retrieval-augmented language modeling framework that treats the language model as a black box and augments it with a tuneable retrieval model.
Outcome:	The proposed framework improves performance on language modeling tasks by 6.3% and 5.1%.

Copied to clipboard

Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, Madian Khabsa

Nearest Neighbor Zero-Shot Inference (2022.emnlp-main)

Challenge:	Existing benchmarks for text comprehension only cover 30 languages, but lack of labeled data is a major obstacle to building functional systems in most languages.
Approach:	They present a multiple-choice machine reading comprehension dataset spanning 122 languages . they use it to evaluate the capabilities of multilingual masked language models and large language models .
Outcome:	The proposed dataset enables the evaluation of text models in high-, medium- and low-resource languages.

Copied to clipboard

Weijia Shi, Julian Michael, Suchin Gururangan, Luke Zettlemoyer

FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation (2023.emnlp-main)

Challenge:	Using non-parametric memory for retrieval-augmented language models yields significant performance boosts over strong zeroshot baselines.
Approach:	They propose a retrieval-augmented language model with fuzzy verbalizers that expands the verbalizes that define different end-task class labels.
Outcome:	The proposed model outperforms non-retrieval-augmented language models on perplexity-based evaluations but gains transfer marginally . the main challenge is to achieve coverage of the verbalizer tokens that define the different end-task class labels.

Copied to clipboard

Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, Hannaneh Hajishirzi

Emerging Cross-lingual Structure in Pretrained Language Models (2020.acl-main)

Challenge:	Evaluating the factuality of long-form text generated by large language models (LMs) is non-trivial because (1) generations often contain a mixture of supported and unsupported pieces of information, making binary judgments of quality inadequate and (2) human evaluation is time-consuming and costly.
Approach:	They introduce a new evaluation that breaks a generation into a series of atomic facts and computes the percentage of atom facts supported by a reliable knowledge source.
Outcome:	The proposed model breaks a generation into atomic facts and computes the percentage of atomic fact supported by a reliable knowledge source.

Copied to clipboard

Alexis Conneau, Shijie Wu, Haoran Li, Luke Zettlemoyer, Veselin Stoyanov

Multilingual Autoregressive Entity Linking (2022.tacl-1)

Challenge:	Recent work has shown that multilingual pretraining works, but is unable to measure these effects.
Approach:	They propose to use multilingual masked language modeling to train a model on concatenated text from multiple languages to find universal latent symmetries in embedding spaces.
Outcome:	The proposed models can be trained on concatenated text from multiple languages without shared vocabulary or domain similarity.

Copied to clipboard

Nicola De Cao, Ledell Wu, Kashyap Popat, Mikel Artetxe, Naman Goyal, Mikhail Plekhanov, Luke Zettlemoyer, Nicola Cancedda, Sebastian Riedel, Fabio Petroni

Knowledge Representation

Challenge:	mGENRE is a sequence-to-sequence system for multilingual entity linking . mGenRE is used to solve language-specific mentions to a multilingual Knowledge Base .
Approach:	They propose a sequence-to-sequence system for multilingual entity linking . they match language-specific mentions against a multilingual Knowledge Base (KB) mGENRE is a sequential system that predicts the name of the target entity token-by-token .
Outcome:	The proposed system improves on three popular MEL benchmarks and shows improvements in accuracy.

CopyBench: Measuring Literal and Non-Literal Reproduction of Copyright-Protected Text in Language Model Generation (2024.emnlp-main)

Copied to clipboard

Tong Chen, Akari Asai, Niloofar Mireshghallah, Sewon Min, James Grimmelmann, Yejin Choi, Hannaneh Hajishirzi, Luke Zettlemoyer, Pang Wei Koh

Breaking the Curse of Multilinguality with Cross-lingual Expert Language Models (2024.emnlp-main)

Challenge:	Existing studies focus on literal copying, but current methods reduce literal copy but not non-literal copying.
Approach:	They propose a benchmark to measure literal and non-literal copying in LMs . they use copyrighted fiction books as text sources to assess literal copying .
Outcome:	The proposed model measures literal and non-literal copying in copyrighted texts . large models show significantly more copying, with literal copying rates increasing .

Copied to clipboard

Terra Blevins, Tomasz Limisiewicz, Suchin Gururangan, Margaret Li, Hila Gonen, Noah Smith, Luke Zettlemoyer

One Embedder, Any Task: Instruction-Finetuned Text Embeddings (2023.findings-acl)

Challenge:	Multilingual language models often underperform monolingual ones due to inter-language competition for model parameters.
Approach:	They propose Cross-lingual Expert Language Models (X-ELM) which mitigates inter-language competition by independently training language models on subsets of the multilingual corpus.
Outcome:	The proposed model outperforms jointly trained multilingual models across all 16 considered languages and transfer the gains to downstream tasks.

Copied to clipboard

Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A. Smith, Luke Zettlemoyer, Tao Yu

BitextEdit: Automatic Bitext Editing for Improved Low-Resource Machine Translation (2022.findings-naacl)

Challenge:	a new method for embedding text is developed for tasks that require specialized encoders . INSTRUCTOR is a single embedder that can generate text embeddables tailored to different tasks and domains based on instruction finetuning .
Approach:	They introduce a new method for computing text embeddings given task instructions . they first annotate instructions for 330 diverse tasks and train INSTRUCTOR on this multitask mixture .
Outcome:	The proposed method improves on 70 embedding evaluation tasks with fewer parameters than the previous best model.

Copied to clipboard

Eleftheria Briakou, Sida Wang, Luke Zettlemoyer, Marjan Ghazvininejad

QuAC: Question Answering in Context (D18-1)

Challenge:	Existing methods to improve Neural Machine Translation (NMT) for lowresource languages are often trained on heuristically aligned or automatically mined data.
Approach:	They propose to filter out imperfect translations that yield unreliable training signals for Neural Machine Translation (NMT) instead, they propose to refine mined bitexts by automatic editing .
Outcome:	The proposed method improves the quality of mined bitexts for low-resource languages by up to 8 BLEU points.

Copied to clipboard

Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, Luke Zettlemoyer

Controlled Crowdsourcing for High-Quality QA-SRL Annotation (2020.acl-main)

Challenge:	a dataset for Question Answering in Context contains 14K information-seeking QA dialogs . questions are often more open-ended, unanswerable, or only meaningful within the dialog context .
Approach:	They propose a dataset for Question Answering in Context that contains 14K dialogs . they use a student to ask questions about a Wikipedia section and a teacher to answer them .
Outcome:	The proposed dataset underperforms humans in a number of reference models . the dataset contains 14K information-seeking dialogs over sections from Wikipedia .

Copied to clipboard

Paul Roit, Ayal Klein, Daniela Stepanov, Jonathan Mamou, Julian Michael, Gabriel Stanovsky, Luke Zettlemoyer, Ido Dagan

Byte Latent Transformer: Patches Scale Better Than Tokens (2025.acl-long)

Challenge:	Question-answer driven Semantic Role Labeling (QA-SRL) is an open and natural flavour of SRL, potentially attainable from laymen.
Approach:	They propose a question-answer driven semantic role labeling approach that uses question-announced questions to label predicate-argument relationships.
Outcome:	The proposed method yields high-quality annotation with dramatically higher coverage, enabling future replicable research of natural semantic annotations.

Copied to clipboard

Artidoro Pagnoni, Ramakanth Pasunuru, Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret Li, Chunting Zhou, Lili Yu, Jason E Weston, Luke Zettlemoyer, Gargi Ghosh, Mike Lewis, Ari Holtzman, Srini Iyer

Green & Sustainable NLP

Challenge:	Existing large language models (LLMs) are trained on bytes, except for tokenization, which groups bytes into a static set of tokens.
Approach:	They propose a new byte-level LLM architecture that encodes bytes into dynamically sized patches, which serve as the primary units of computation.
Outcome:	The proposed architecture matches tokenization-based models with improvements in inference efficiency and robustness.

Getting MoRE out of Mixture of Language Model Reasoning Experts (2023.findings-emnlp)

Copied to clipboard

Chenglei Si, Weijia Shi, Chen Zhao, Luke Zettlemoyer, Jordan Boyd-Graber

NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating System (L18-1)

Challenge:	Existing large language models (LLMs) have poor generalizability on question types beyond those seen in the prompt.
Approach:	They propose a framework that integrates specialized language models to generalize across question types that require distinct reasoning abilities.
Outcome:	The proposed framework gives higher accuracy than any single specialized model on a collection of 12 QA datasets from four reasoning types.

Copied to clipboard

Xi Victoria Lin, Chenglong Wang, Luke Zettlemoyer, Michael D. Ernst

Challenge:	NL2Bash is a new semantic parsing problem for mapping English sentences to Bash commands.
Approach:	They propose a dataset of English commands and expert-written Bash commands to map English sentences to Bash.
Outcome:	The proposed methods are significantly larger (from two to ten times) than most existing benchmarks.

Better Alignment with Instruction Back-and-Forth Translation (2024.findings-emnlp)

Copied to clipboard

Thao Nguyen, Jeffrey Li, Sewoong Oh, Ludwig Schmidt, Jason Weston, Luke Zettlemoyer, Xian Li

Better Character Language Modeling through Morphology (P19-1)

Challenge:	et al., 2023) proposes a method to improve instruction-tuning data . e.g., we generate synthetic instructions using the backtranslation approach .
Approach:	They propose a method to improve instruction-tuning data using web-based inputs . they generate synthetic instructions using the backtranslation approach and filter the generated data .
Outcome:	The proposed method improves the quality of instruction-tuning data based on preprocessed texts . it yields better AlpacaEval win rates than direct distillation .

Copied to clipboard

Terra Blevins, Luke Zettlemoyer

Scalable Zero-shot Entity Linking with Dense Entity Retrieval (2020.emnlp-main)

Challenge:	Inflected words benefit more from explicitly modeling morphology than uninflectes . morphological supervision is also used to augment character language models in low-resource languages .
Approach:	They add morphological supervision to character language models via multitasking to improve BPC performance across 24 languages even when morphology data and language modeling data are disjointed.
Outcome:	The addition improves performance even when morphology data and language modeling data are disjointed.

Copied to clipboard

Ledell Wu, Fabio Petroni, Martin Josifoski, Sebastian Riedel, Luke Zettlemoyer

Revisiting Machine Translation for Cross-lingual Classification (2023.emnlp-main)

Challenge:	Existing methods for entity linking use manually curated mention tables and incoming Wikipedia link popularity.
Approach:	They propose a BERT-based entity linking model with a bi-encoder that embeds the mention context and the entity descriptions and then re-ranked the candidate with . they also evaluate the accuracy-speed trade-off inherent to large pre-trained models.
Outcome:	The proposed model is state-of-the-art on recent zero-shot benchmarks and established non-zero-shot evaluations.

Copied to clipboard

Mikel Artetxe, Vedanuj Goswami, Shruti Bhosale, Angela Fan, Luke Zettlemoyer

Altogether: Image Captioning via Re-aligning Alt-text (2024.emnlp-main)

Challenge:	Recent work in cross-lingual learning has pivoted around multilingual models, which are typically pretrained on unlabeled corpora in multiple languages using some form of language modeling objective.
Approach:	They propose to use a stronger machine translation system to mitigat mismatch between training on original text and running inference on machine translated text.
Outcome:	The proposed approach is highly task dependent and calls into question the dominance of multilingual models for cross-lingual classification.

Copied to clipboard

Hu Xu, Po-Yao Huang, Xiaoqing Tan, Ching-Feng Yeh, Jacob Kahn, Christine Jou, Gargi Ghosh, Omer Levy, Luke Zettlemoyer, Wen-tau Yih, Shang-Wen Li, Saining Xie, Christoph Feichtenhofer

Multimodality

Challenge:	Existing captioning models ignore existing alt-text metadata and lack transparency if training data is unknown.
Approach:	They propose an approach to edit and re-align alt-texts associated with images using human annotation.
Outcome:	The proposed approach improves image captions and improves text-to-image generation and zero-shot image classification tasks.

Adversarial Example Generation with Syntactically Controlled Paraphrase Networks (N18-1)

Copied to clipboard

Mohit Iyyer, John Wieting, Kevin Gimpel, Luke Zettlemoyer

VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding (2021.emnlp-main)

Challenge:	Existing approaches to learn to do syntactically controlled paraphrase generation are limited . lexical, pragmatic, and syntaktic variation can hurt generalization of models trained on them .
Approach:	They propose a new approach for learning to do syntactically controlled paraphrase generation using a parser.
Outcome:	The proposed model generates paraphrases that follow their target specifications without decreasing paraphrase quality compared to baseline models . it improves the robustness of the models to syntactic variation when used to augment training data.

Copied to clipboard

Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, Christoph Feichtenhofer

Multimodality

Challenge:	Recent work adopts a "pre-training + fine-tuning" approach for zero-shot transfer to end tasks without fine- tuning.
Approach:	They propose a contrastive approach to pre-train a transformer model for zero-shot video and text understanding without using any labels on downstream tasks.
Outcome:	The proposed model outperforms supervised approaches on downstream tasks and outperformed previous approaches.

BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension (2020.acl-main)

Copied to clipboard

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, Luke Zettlemoyer

Moving Down the Long Tail of Word Sense Disambiguation with Gloss Informed Bi-encoders (2020.acl-main)

Challenge:	Recent work has shown gains by improving the distribution of masked tokens and the order in which mucked tokens are predicted.
Approach:	They propose a denoising autoencoder for pretraining sequence-to-sequence models that corrupts text with an arbitrary noising function and learns a model to reconstruct the original text.
Outcome:	The proposed model outperforms RoBERTa on GLUE and SQUAD and provides a 1.1 BLEU increase over a back-translation system for machine translation.

Copied to clipboard

Terra Blevins, Luke Zettlemoyer

In-context Examples Selection for Machine Translation (2023.findings-acl)

Challenge:	Existing models for Word Sense Disambiguation are not uniformly distributed on rare or unseen senses.
Approach:	They propose a bi-encoder model that embeds the target word with its context and the dictionary definition, or gloss, of each sense.
Outcome:	The proposed model outperforms previous state-of-the-art models on English all-words WSD, with a 31.1% error reduction on less frequent senses over prior work.

Copied to clipboard

Sweta Agrawal, Chunting Zhou, Mike Lewis, Luke Zettlemoyer, Marjan Ghazvininejad

Machine Translation

Challenge:	Large-scale generative models can perform a wide range of NLP tasks using in-context learning.
Approach:	They aim to understand the properties of good in-context examples for machine translation in both in-domain and out-of-domain settings.
Outcome:	The proposed model outperforms a strong kNN-MT baseline in 2 out of 4 out-of-domain datasets.

UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models (2022.emnlp-main)

Copied to clipboard

Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I. Wang, Victor Zhong, Bailin Wang, Chengzu Li, Connor Boyle, Ansong Ni, Ziyu Yao, Dragomir Radev, Caiming Xiong, Lingpeng Kong, Rui Zhang, Noah A. Smith, Luke Zettlemoyer, Tao Yu

Active Learning for Coreference Resolution using Discrete Annotation (2020.acl-main)

Challenge:	Structured knowledge grounding (SKG) uses structured knowledge to complete user requests . since inputs and outputs of SKG tasks are heterogeneous, they have been studied separately .
Approach:	They propose a framework that unifies 21 SKG tasks into a text-to-text format . they use unifiedSKG to benchmark T5 with different sizes .
Outcome:	The proposed framework unifies 21 SKG tasks into a text-to-text format . it achieves state-of-the-art performance on almost all of the 21 tasks, the authors show .

Copied to clipboard

Belinda Z. Li, Gabriel Stanovsky, Luke Zettlemoyer

Mask-Predict: Parallel Decoding of Conditional Masked Language Models (D19-1)

Challenge:	Exhaustively annotating coreference is expensive as it requires tracking coreference chains across long passages of text.
Approach:	They propose a pairwise annotation technique which asks annotators to identify mention antecedents if a presented mention pair is not coreferent.
Outcome:	The proposed method is much more efficient when combined with a mention clustering algorithm for selecting which examples to label . future work can use the proposed protocol to develop coreference models for new domains.

Copied to clipboard

Marjan Ghazvininejad, Omer Levy, Yinhan Liu, Luke Zettlemoyer

BERT for Coreference Resolution: Baselines and Analysis (D19-1)

Challenge:	a masked language model is used to train a model to predict subsets of mangled words . a parallel decoding algorithm can be used to generate translations in a constant number of iterations.
Approach:	They propose a model and a parallel decoding algorithm which train a machine to predict any subset of target words . they introduce conditional masked language models (CMLMs) which are trained with a mangled language model objective .
Outcome:	The proposed model improves state-of-the-art performance levels for non-autoregressive and parallel decoding models by over 4 BLEU on average.

Copied to clipboard

Mandar Joshi, Omer Levy, Luke Zettlemoyer, Daniel Weld

Noisy Channel Language Model Prompting for Few-Shot Text Classification (2022.acl-long)

Challenge:	Recent BERT-based models have reported dramatic gains on multiple semantic benchmarks including question-answering, natural language inference, and named entity recognition.
Approach:	They apply BERT to coreference resolution, achieving a new state of the art on the GAP and OntoNotes benchmarks.
Outcome:	A qualitative analysis of model predictions shows that BERT-large is better at distinguishing between related but distinct entities, but there is room for improvement in modeling document-level context, conversations, and mention paraphrasing.

Copied to clipboard

Sewon Min, Mike Lewis, Hannaneh Hajishirzi, Luke Zettlemoyer

Nicholas FitzGerald, Julian Michael, Luheng He, Luke Zettlemoyer

Challenge:	Prior work has suggested methods for finding better prompt or scoring of the output from the model.
Approach:	They propose a noisy channel approach for language model prompting in few-shot text classification by in-context demonstration or prompt tuning.
Outcome:	The proposed model outperforms direct models in both demonstration and prompt tuning.

Large-Scale QA-SRL Parsing (P18-1)

Copied to clipboard

AmbigQA: Answering Ambiguous Open-domain Questions (2020.emnlp-main)

Challenge:	a crowd-sourced approach to learning semantic parsers to predict predicateargument structures is open to many researchers.
Approach:	They propose a large-scale corpus of Question-Answer driven Semantic Role Labeling annotations . they also propose QA-SRL Bank 2.0, a crowd-sourcing scheme that can be used to train high quality parsers .
Outcome:	The proposed QA-SRL parser can generate high-quality questions at low cost and is intuitive to non-experts.

Copied to clipboard

Sewon Min, Julian Michael, Hannaneh Hajishirzi, Luke Zettlemoyer

Question Answering

Challenge:	Existing open-domain question answering systems assume questions have a single welldefined answer.
Approach:	They propose an open-domain question answering task which involves finding every plausible answer and rewriting the question for each one to resolve the ambiguity.
Outcome:	The proposed task is based on a dataset covering 14,042 open-domain questions . it shows that strong models benefit from weakly supervised learning .

Iterative Search for Weakly Supervised Semantic Parsing (N19-1)

Copied to clipboard

Pradeep Dasigi, Matt Gardner, Shikhar Murty, Luke Zettlemoyer, Eduard Hovy

Learning Programmatic Idioms for Scalable Semantic Parsing (D19-1)

Challenge:	Recent work has focused on training semantic parsers via weak supervision from denotations alone.
Approach:	They propose an iterative training algorithm that alternates between searching for consistent logical forms and maximizing the marginal likelihood of the retrieved ones.
Outcome:	The proposed algorithm outperforms the previous best systems on WikiTableQuestions and Cornell Natural Language Visual Reasoning (NLVR) iteratively train models that provide guidance to subsequent models to search for logical forms of increasing complexity, thus dealing with spuriousness.

Copied to clipboard

Srinivasan Iyer, Alvin Cheung, Luke Zettlemoyer

Multimodality

Challenge:	In state-of-the-art semantic parsers map natural language instructions to source code . idioms improve the accuracy of semantic parses, allowing for faster decoding .
Approach:	They propose an iterative method to extract code idioms from large source code corpora . they use most-frequent subtrees of their syntax trees to train semantic parsers to apply them .
Outcome:	The proposed method improves the state-of-the-art semantic parsers' accuracy and training time by more than 50%.

Cloze-driven Pretraining of Self-attention Networks (D19-1)

Copied to clipboard

Alexei Baevski, Sergey Edunov, Yinhan Liu, Luke Zettlemoyer, Michael Auli

Grounded Adaptation for Zero-shot Executable Semantic Parsing (2020.emnlp-main)

Challenge:	Existing work on pretraining language models has used unidirectional (left-to-right) or bi-directional (both left-to right and right-to left) LMs with loss function.
Approach:	They propose a bi-directional transformer model that pretrains both directions of a large language-model-inspired self-attention cloze model and propose clozing to predict each word in the training data.
Outcome:	The proposed model performs well on GLUE and state of the art benchmarks consistent with BERT.

Copied to clipboard

Victor Zhong, Mike Lewis, Sida I. Wang, Luke Zettlemoyer

E3: Entailment-driven Extracting and Editing for Conversational Machine Reading (P19-1)

Challenge:	Existing semantic parsers are usually engineered for each application environment, but they struggle when deployed to a new database.
Approach:	They propose a method to adapt existing semantic parsers to new environments . they propose combining a forward semantic parsed with a backward utterance generator to synthesize data in the new environment and select cycle-consistent examples to adapt the parser.
Outcome:	The proposed procedure outperforms data-augmentation and improves execution accuracy on the Spider, Sparc, and CoSQL zero-shot semantic parsing tasks.

Copied to clipboard

Victor Zhong, Luke Zettlemoyer

Neural Metaphor Detection in Context (D18-1)

Challenge:	Conversational machine reading systems help users answer high-level questions when they do not know the exact rules by which the decision is made.
Approach:	They propose a conversational machine reading model that extracts a set of decision rules from a procedural text which the system must read to figure out what to ask the user.
Outcome:	The proposed model outperforms existing systems and a BERT-based baseline on the ShARC conversational machine reading dataset and provides an explainable alternative to prior work.

Copied to clipboard

Ge Gao, Eunsol Choi, Yejin Choi, Luke Zettlemoyer

Sentiment Analysis

Challenge:	Existing models focus on limited forms of linguistic context, such as unigrams.
Approach:	They propose end-to-end neural models for detecting metaphorical word use in context . they show that bi-directional biLSTM models which operate on complete sentences work well .
Outcome:	The proposed models show that they can learn rich contextual word representations . they are compared to previous models which focused on limited linguistic context .

Question Answering Infused Pre-training of General-Purpose Contextualized Representations (2022.findings-acl)

Copied to clipboard

Robin Jia, Mike Lewis, Luke Zettlemoyer

Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? (2022.emnlp-main)

Challenge:	Existing pretraining objectives for question answering (QA) are not optimized for being immediately useful without fine-tuning.
Approach:	They propose a pre-training objective based on question answering (QA) that is based more directly on context.
Outcome:	The proposed model matches predictions of a more accurate cross-encoder model on 80 million synthesized QA pairs and achieves large improvements over previous state-of-the-art models on paraphrase detection and fewshot named entity recognition.

Copied to clipboard

Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, Luke Zettlemoyer

Detecting Hallucinated Content in Conditional Neural Sequence Generation (2021.findings-acl)

Challenge:	Large language models can in-context learn by conditioning on a few input-label pairs and making predictions for new inputs.
Approach:	They propose to use ground truth demonstrations to replace labels in demonstrations . they also show that other aspects of the demonstrations are key drivers of endtask performance .
Outcome:	The proposed model outperforms zeroshot inference on a wide range of tasks using ground truth demonstrations.

Copied to clipboard

Chunting Zhou, Graham Neubig, Jiatao Gu, Mona Diab, Francisco Guzmán, Luke Zettlemoyer, Marjan Ghazvininejad

Matt Gardner, Pradeep Dasigi, Srinivasan Iyer, Alane Suhr, Luke Zettlemoyer

Challenge:	Neural sequence models can generate fluent sentences, but they can also hallucinate additional content not supported by the input.
Approach:	They propose a task to predict whether each token in the output sequence is hallucinated and collect manually annotated evaluation sets for this task.
Outcome:	The proposed method outperforms baseline methods on machine translation and abstractive summarization datasets and achieves significant improvements in both supervised and unsupervised settings.

Neural Semantic Parsing (P18-5)

Copied to clipboard

Toward Human Readable Prompt Tuning: Kubrick’s The Shining is a good movie, and a good prompt too? (2023.findings-emnlp)

Challenge:	Semantic parsing is the study of translating natural language utterances into machine-executable programs.
Approach:	They will describe the various approaches researchers have taken to translate natural language into a formal language . they will also discuss why much recent work has chosen to use standard programming languages instead of more linguistically-motivated representations.
Outcome:	This paper will describe the various approaches researchers have taken to translate natural language into a formal language.

Copied to clipboard

Weijia Shi, Xiaochuang Han, Hila Gonen, Ari Holtzman, Yulia Tsvetkov, Luke Zettlemoyer

RoMQA: A Benchmark for Robust, Multi-evidence, Multi-answer Question Answering (2023.findings-emnlp)

Challenge:	Large language models can perform downstream tasks in a zero-shot fashion, given natural language prompts that specify the desired behavior.
Approach:	They propose a human readable prompt tuning method that incorporates a fluency constraint to find a distribution of effective and fluent prompts.
Outcome:	The proposed method outperforms baselines by 7.0% across three tasks.

Copied to clipboard

Victor Zhong, Weijia Shi, Wen-tau Yih, Luke Zettlemoyer

Surface Form Competition: Why the Highest Probability Answer Isn’t Always Right (2021.emnlp-main)

Challenge:	Existing QA models are not robust to variations in question constraints, but can be made more robust by tuning on clusters of related questions.
Approach:	They introduce RoMQA, the first benchmark for robust, multi-evidence, multianswer question answering (QA) RoMQ contains clusters of related questions that are derived from the Wikidata knowledge graph .
Outcome:	The proposed model is the first benchmark for robust, multi-evidence, multianswer question answering (QA) compared to prior QA datasets, it has more human-written questions that require reasoning over more evidence text and have, on average, many more correct answers.

Copied to clipboard

Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, Luke Zettlemoyer

Long Short-Term Memory as a Dynamically Computed Element-wise Weighted Sum (P18-2)

Challenge:	Large language models have shown promising results in zero-shot settings due to surface form competition . since probability mass is finite, this lowers the probability of the correct answer .
Approach:	They propose a scoring function that compensates for surface form competition by reweighing each option according to its a priori likelihood.
Outcome:	The proposed scoring function achieves consistent gains in zero-shot over calibrated and uncalibrated scoring functions on all GPT-2 and GPT-3 models on a variety of multiple choice datasets.

Copied to clipboard

Omer Levy, Kenton Lee, Nicholas FitzGerald, Luke Zettlemoyer

Training Trajectories of Language Models Across Scales (2023.acl-long)

Challenge:	LSTMs were introduced to combat vanishing gradients in simple RNNs by augmenting them with gated additive recurrent connections.
Approach:	They propose to decouple the LSTM’s gates from the embedded RNN and create a new class of RNNs where the recurrence computes an element-wise weighted sum of context-independent functions of the input.
Outcome:	The proposed model performs as well as an LSTM on a range of problems, strongly suggesting that the gates are doing much more in practice than just alleviating vanishing gradients.

Copied to clipboard

Mengzhou Xia, Mikel Artetxe, Chunting Zhou, Xi Victoria Lin, Ramakanth Pasunuru, Danqi Chen, Luke Zettlemoyer, Veselin Stoyanov

Jointly Predicting Predicates and Arguments in Neural Semantic Role Labeling (P18-2)

Challenge:	Scaling up language models has led to unprecedented performance gains, but little is understood about how the training dynamics change as models get larger.
Approach:	They analyze the training checkpoints of different-sized OPT models on next-token prediction, sequence-level generation and downstream tasks.
Outcome:	The results show that language models of different sizes learn more during training . small models halt at hallucinations, larger ones learn to assign lower probabilities .

Copied to clipboard

Luheng He, Kenton Lee, Omer Levy, Luke Zettlemoyer

Semantic Parsing

Challenge:	Recent models that use gold predicates only use a single predicate at a time.
Approach:	They propose an end-to-end approach for jointly predicting all predicates, arguments spans, and the relations between them.
Outcome:	The proposed model can model overlapping spans across different predicates in the same output structure without gold predicate predications.

Mapping Language to Code in Programmatic Context (D18-1)

Copied to clipboard

Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Luke Zettlemoyer

Programming Languages in NLP

Challenge:	Existing approaches for automatically mapping natural language to executable code have considered limited language or code environments.
Approach:	They propose a task of generating class member functions given English documentation and the programmatic context provided by the rest of the class.
Outcome:	The proposed model can generate member functions from documentation and the class environment.

Higher-Order Coreference Resolution with Coarse-to-Fine Inference (N18-2)

Copied to clipboard

Kenton Lee, Luheng He, Luke Zettlemoyer

Demystifying Prompts in Language Models via Perplexity Estimation (2023.findings-emnlp)

Challenge:	a new approach to coreference resolution uses a span-ranking architecture as an attention mechanism to iteratively refine span representations.
Approach:	They propose a fully-differentiable approximation to higher-order inference for coreference resolution . they propose introducing a coarse-to-fine approach that incorporates a less accurate but more efficient bilinear factor .
Outcome:	The proposed model significantly improves accuracy on the English OntoNotes benchmark while being far more computationally efficient.

Copied to clipboard

Hila Gonen, Srini Iyer, Terra Blevins, Noah Smith, Luke Zettlemoyer

M2D2: A Massively Multi-Domain Language Modeling Dataset (2022.emnlp-main)

Challenge:	Language models can be prompted to perform a wide variety of tasks with zero- and few-shot learning.
Approach:	They propose a method to automatically extend a small seed set of manually written prompts by paraphrasing with GPT3 and backtranslation.
Outcome:	The proposed method extends a small seed set of manually written prompts by paraphrasing with GPT3 and backtranslation.

Copied to clipboard

Machel Reid, Victor Zhong, Suchin Gururangan, Luke Zettlemoyer