Papers by Wen-tau Yih

Policy Shaping and Generalized Update Equations for Semantic Parsing from Denotations (D18-1)

Challenge:	Existing models for entity linking are limited to entity disambiguation and require mention boundaries to be given in the input.
Approach:	They propose a fast end-to-end entity linking model that uses a biencoder to jointly detect mentions and link in one pass.
Outcome:	The proposed model outperforms the current state of the art on WebQSP and GraphQuestions with extended annotations that cover multiple entities per question.

Copied to clipboard

Dipendra Misra, Ming-Wei Chang, Xiaodong He, Wen-tau Yih

Improving Factuality with Explicit Working Memory (2025.acl-long)

Challenge:	Existing learning approaches for parsing from denotations (SpFD) do not provide access to correct representations, so there are two steps for every training example.
Approach:	They propose a framework for parsing from denotations that generalizes three different learning algorithms.
Outcome:	The proposed framework outperforms previous work by 5.0% absolute on exact match accuracy on a question answering dataset.

Copied to clipboard

Mingda Chen, Yang Li, Karthik Padthe, Rulin Shao, Alicia Yi Sun, Luke Zettlemoyer, Gargi Ghosh, Wen-tau Yih

Information Retrieval

Challenge:	Large language models can generate factually inaccurate content, a problem known as hallucination.
Approach:	They propose an approach that integrates a working memory that receives feedback from external resources.
Outcome:	The proposed method outperforms baselines on four fact-seeking datasets and increases the factuality metric by 2 to 6 points absolute.

Blockwise Self-Attention for Long Document Understanding (2020.findings-emnlp)

Copied to clipboard

Jiezhong Qiu, Hao Ma, Omer Levy, Wen-tau Yih, Sinong Wang, Jie Tang

Nonparametric Masked Language Modeling (2023.findings-acl)

Challenge:	Recent advances in pre-training and fine-tuning methods have drastically reshaped the landscape of natural language processing research.
Approach:	They propose a lightweight BERT model that introduces sparse block structures into the attention matrix to reduce memory consumption and training/inference time.
Outcome:	The proposed model uses 18.7-36.1% less memory and 12.0-25.1% more time to learn compared to an advanced BERT-based model, RoBERTa.

Copied to clipboard

Sewon Min, Weijia Shi, Mike Lewis, Xilun Chen, Wen-tau Yih, Hannaneh Hajishirzi, Luke Zettlemoyer

Dissecting Contextual Word Embeddings: Architecture and Representation (D18-1)

Challenge:	Existing language models (LMs) predict tokens with a softmax over a finite vocabulary, which can make it difficult to predict rare tokens or phrases.
Approach:	They introduce a nonparametric masked language model that replaces a softmax with a distribution over every phrase in a reference corpus and uses an in-batch approximation to train it.
Outcome:	The proposed model outperforms larger parametric models on 16 tasks including classification, fact probing and question answering.

Copied to clipboard

Matthew E. Peters, Mark Neumann, Luke Zettlemoyer, Wen-tau Yih

RECONSIDER: Improved Re-Ranking using Span-Focused Cross-Attention for Open Domain Question Answering (2021.naacl-main)

Challenge:	Existing work on learning contextual representations has used LSTM-based biLMs, but there is no reason to believe this is effective.
Approach:	They propose to use pre-trained bidirectional language models to learn contextual word embeddings for four NLP tasks and to use them to study the effects of architecture on endtask accuracy.
Outcome:	The proposed models outperform word embeddings for four NLP tasks and all learn representations that vary with network depth.

Copied to clipboard

Srinivasan Iyer, Sewon Min, Yashar Mehdad, Wen-tau Yih

Natural Language to Structured Query Generation via Meta-Learning (N18-2)

Challenge:	State-of-the-art Machine Reading Comprehension (MRC) models for Open-domain Question Answering (QA) achieve high recall amongst top few predictions, but low overall accuracy, motivating the need for answer re-ranking.
Approach:	They propose a method to make answer re-ranking successful for span-extraction tasks even beyond large pre-training.
Outcome:	The proposed approach achieves 45.5% Exact Match accuracy on Natural Questions and 61.7% on TriviaQA.

Copied to clipboard

Po-Sen Huang, Chenglong Wang, Rishabh Singh, Wen-tau Yih, Xiaodong He

Text Generation

Challenge:	Conventional supervised training is a pervasive paradigm for NLP problems . however, examples of the same problem may vary widely . a few-shot meta-learning scenario is used to learn multiple models .
Approach:	They propose a learning protocol that treats each example as a unique pseudo-task . they use a few-shot meta-learning scenario to reduce the original learning problem to a single example .
Outcome:	The proposed learning protocol achieves 1.1%–5.4% accuracy gains over non-meta-learning counterparts on a WikiSQL dataset.

Trusting Your Evidence: Hallucinate Less with Context-aware Decoding (2024.naacl-short)

Copied to clipboard

Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, Wen-tau Yih

TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data (2020.acl-main)

Challenge:	Language models (LMs) often struggle to pay enough attention to the input context, and generate texts that are unfaithful or contain hallucinations.
Approach:	They propose a context-aware decoding technique that amplifies the difference between the output probabilities when a model is used with and without context.
Outcome:	The proposed model significantly improves faithfulness of different LM families including OPT, GPT, LLaMA, and FLAN-T5 for summarization tasks.

Copied to clipboard

Pengcheng Yin, Graham Neubig, Wen-tau Yih, Sebastian Riedel

CITADEL: Conditional Token Interaction via Dynamic Lexical Routing for Efficient and Effective Multi-Vector Retrieval (2023.acl-long)

Challenge:	Recent years have witnessed the burgeoning of pretrained language models (LMs) for text-based natural language understanding tasks.
Approach:	They propose a pretrained language model that jointly learns representations for NL sentences and (semi-)structured tables.
Outcome:	The proposed model performs best on the weakly-supervised semantic parsing benchmark WikiTableQuestions while performing competitively on the text-to-SQL dataset Spider.

Copied to clipboard

Minghan Li, Sheng-Chieh Lin, Barlas Oguz, Asish Ghoshal, Jimmy Lin, Yashar Mehdad, Wen-tau Yih, Xilun Chen

Green & Sustainable NLP

Challenge:	Existing multi-vector retrieval methods are slower and require more space to store indices compared to their single-vektor counterparts.
Approach:	They propose a multi-vector retrieval method that uses dynamic lexical routing to route different token vectors to the predicted lexicals.
Outcome:	The proposed method achieves state-of-the-art performance on several benchmark datasets while being nearly 40 times faster than the current state-out-of the-art method.

Expand, Rerank, and Retrieve: Query Reranking for Open-Domain Question Answering (2023.findings-acl)

Copied to clipboard

Yung-Sung Chuang, Wei Fang, Shang-Wen Li, Wen-tau Yih, James Glass

Information Retrieval

Challenge:	Empirically, EAR improves top-5/20 accuracy by 3-8 and 5-10 points . dense retrievers are limited by their inability to perform semantic matching for relevant passages that have low lexical overlap with the query.
Approach:	They propose a query expansion and reranking approach for improving passage retrieval with the application to open-domain question answering.
Outcome:	Empirically, EAR improves top-5/20 accuracy by 3-8 and 5-10 points when compared to a vanilla query expansion model and a dense retrieval model.

Knowledge Extraction on Semi-Structured Content: Does It Remain Relevant for Question Answering in the Era of LLMs? (2026.eacl-long)

Copied to clipboard

Kai Sun, Yin Huang, Srishti Mehra, Mohammad Kachuee, Xilun Chen, Renjie Tao, Zhaojiang Lin, Andrea Jessee, Nirav Shah, Alex L Betty, Yue Liu, Anuj Kumar, Wen-tau Yih, Xin Luna Dong

Information Extraction & Text Mining

Challenge:	Existing literature on knowledge extraction for question answering questions whether it is still relevant for question answerrs.
Approach:	They extend an existing benchmark with knowledge extraction annotations and evaluate commercial and open-source LLMs of varying sizes.
Outcome:	The proposed model can achieve high QA accuracy, but can still benefit from knowledge extraction through augmentation with extracted triples and multi-task learning.

Improving Passage Retrieval with Zero-Shot Question Generation (2022.emnlp-main)

Copied to clipboard

Devendra Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan, Wen-tau Yih, Joelle Pineau, Luke Zettlemoyer

Low-Resource NLP

Challenge:	Existing re-ranking methods for open-domain question answering are not domain- or task-specific.
Approach:	They propose a simple and effective re-ranking method for improving passage retrieval in open-domain question answering.
Outcome:	The proposed method outperforms strong supervised models on open-domain questions and triviaQA datasets on top-1000 passages.

REPLUG: Retrieval-Augmented Black-Box Language Models (2024.naacl-long)

Copied to clipboard

Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Richard James, Mike Lewis, Luke Zettlemoyer, Wen-tau Yih

Open-Domain Question Answering (2020.acl-tutorials)

Challenge:	Existing retrieval-augmented language models require access to internal representations to enhance performance.
Approach:	They introduce a retrieval-augmented language modeling framework that treats the language model as a black box and augments it with a tuneable retrieval model.
Outcome:	The proposed framework improves performance on language modeling tasks by 6.3% and 5.1%.

Copied to clipboard

Danqi Chen, Wen-tau Yih

Few-Shot Data Synthesis for Open Domain Multi-Hop Question Answering (2024.eacl-long)

Challenge:	tutorial provides a comprehensive overview of cutting-edge research in open-domain question answering (QA)
Approach:	tutorial provides a comprehensive overview of cutting-edge research in open-domain question answering . focus will shift to cutting- edge models proposed for open- domain QA .
Outcome:	The tutorial will cover cutting-edge research in open-domain question answering (QA) it will cover two-stage retriever-reader approaches, dense retriever and end-to-end training, and retriever free methods .

Copied to clipboard

Mingda Chen, Xilun Chen, Wen-tau Yih

FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation (2023.emnlp-main)

Challenge:	Recent approaches to multi-hop question answering rely on in-context learning . however, these models contain billions of parameters making them inefficient at inference time.
Approach:	They propose a framework that allows for improving smaller language models with less than 10 human-annotated QA pairs by synthesizing millions of multi-hop questions and claims to fine tune language models.
Outcome:	The proposed framework improves model performance on multi-hop question answering and fact verification benchmarks while being almost one-third the size in parameter count.

Copied to clipboard

Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, Hannaneh Hajishirzi

Salient Phrase Aware Dense Retrieval: Can a Dense Retriever Imitate a Sparse One? (2022.findings-emnlp)

Challenge:	Evaluating the factuality of long-form text generated by large language models (LMs) is non-trivial because (1) generations often contain a mixture of supported and unsupported pieces of information, making binary judgments of quality inadequate and (2) human evaluation is time-consuming and costly.
Approach:	They introduce a new evaluation that breaks a generation into a series of atomic facts and computes the percentage of atom facts supported by a reliable knowledge source.
Outcome:	The proposed model breaks a generation into atomic facts and computes the percentage of atomic fact supported by a reliable knowledge source.

Copied to clipboard

Xilun Chen, Kushal Lakhotia, Barlas Oguz, Anchit Gupta, Patrick Lewis, Stan Peshterliev, Yashar Mehdad, Sonal Gupta, Wen-tau Yih

Information Retrieval

Challenge:	Existing sparse retrievers lack the ability to match salient phrases and rare entities in the query.
Approach:	They introduce a dense Lexical Model that can be trained to imitate a sparse one.
Outcome:	The proposed model outperforms sparse retrievers on a range of tasks including five question answering datasets and the MS MARCO passage retrieval.

Be Consistent! Improving Procedural Text Comprehension using Label Consistency (N19-1)

Copied to clipboard

Xinya Du, Bhavana Dalvi, Niket Tandon, Antoine Bosselut, Wen-tau Yih, Peter Clark, Claire Cardie

Challenge:	Existing systems for procedural text comprehension still struggle with this task . evaluative work shows that consistent predictions from multiple entities can improve performance .
Approach:	They propose a framework that leverages label consistency during training to improve prediction performance.
Outcome:	The proposed framework significantly improves prediction performance over previous state-of-the-art systems on a standard benchmark dataset for procedural text, ProPara.

An Imitation Game for Learning Semantic Parsers from User Interaction (2020.emnlp-main)

Copied to clipboard

Ziyu Yao, Yiqi Tang, Wen-tau Yih, Huan Sun, Yu Su

Challenge:	Existing methods for learning semantic parsers are expensive and tedious . despite the widespread applications, bootstrapping and fine-tuning is tedious a task .
Approach:	They propose an alternative method for learning semantic parsers directly from users . they propose an annotation-efficient imitation learning algorithm that iteratively collects new datasets .
Outcome:	The proposed method is cost-effective and shows promising performance on the text-to-SQL problem.

Task-aware Retrieval with Instructions (2023.findings-acl)

Copied to clipboard

Akari Asai, Timo Schick, Patrick Lewis, Xilun Chen, Gautier Izacard, Sebastian Riedel, Hannaneh Hajishirzi, Wen-tau Yih

Low-Resource NLP

Challenge:	Existing models that learn intents from labeled data are complicated and require a vast number of annotated examples to train a model.
Approach:	They propose a general-purpose task-aware retrieval system with instructions that can adapt to a new task without any parameter updates.
Outcome:	The proposed system outperforms two benchmarks on a set of domains and tasks on X2-Retrieval.

One Embedder, Any Task: Instruction-Finetuned Text Embeddings (2023.findings-acl)

Copied to clipboard

Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A. Smith, Luke Zettlemoyer, Tao Yu

Dense Passage Retrieval for Open-Domain Question Answering (2020.emnlp-main)

Challenge:	a new method for embedding text is developed for tasks that require specialized encoders . INSTRUCTOR is a single embedder that can generate text embeddables tailored to different tasks and domains based on instruction finetuning .
Approach:	They introduce a new method for computing text embeddings given task instructions . they first annotate instructions for 330 diverse tasks and train INSTRUCTOR on this multitask mixture .
Outcome:	The proposed method improves on 70 embedding evaluation tasks with fewer parameters than the previous best model.

Copied to clipboard

Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, Wen-tau Yih

PrismRAG: Boosting RAG Factuality with Distractor Resilience and Strategized Reasoning (2025.emnlp-industry)

Challenge:	Open-domain question answering relies on efficient passage retrieval to select candidate contexts.
Approach:	They propose a dual-encoder framework that can be implemented to retrieve passages from a small number of questions and passages.
Outcome:	The proposed system outperforms a strong Lucene-BM25 system in top-20 passage retrieval accuracy on multiple open-domain QA benchmarks.

Copied to clipboard

Mohammad Kachuee, Teja Gollapudi, Minseok Kim, Yin Huang, Kai Sun, Xiao Yang, Jiaqi Wang, Nirav Shah, Yue Liu, Aaron Colak, Anuj Kumar, Wen-tau Yih, Xin Luna Dong

Reasoning

Challenge:	Existing methods to improve factuality of large language models (LLMs) rely on human-engineered instructions.
Approach:	They propose a retrieval-augmented generation framework that trains the model with distractor-aware QA pairs mixing gold evidence with subtle distractor passages and instills reasoning-centric habits that make the LLM plan, rationalize, and synthesize without extensive human engineered instructions.
Outcome:	The proposed framework outperforms state-of-the-art solutions across 12 open-book RAG QA benchmarks and is being deployed in production.

QuAC: Question Answering in Context (D18-1)

Copied to clipboard

Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, Luke Zettlemoyer

Instruction-tuned Language Models are Better Knowledge Learners (2024.acl-long)

Challenge:	a dataset for Question Answering in Context contains 14K information-seeking QA dialogs . questions are often more open-ended, unanswerable, or only meaningful within the dialog context .
Approach:	They propose a dataset for Question Answering in Context that contains 14K dialogs . they use a student to ask questions about a Wikipedia section and a teacher to answer them .
Outcome:	The proposed dataset underperforms humans in a number of reference models . the dataset contains 14K information-seeking dialogs over sections from Wikipedia .

Copied to clipboard

Zhengbao Jiang, Zhiqing Sun, Weijia Shi, Pedro Rodriguez, Chunting Zhou, Graham Neubig, Xi Lin, Wen-tau Yih, Srini Iyer

ImpRAG: Retrieval-Augmented Generation with Implicit Queries (2025.findings-emnlp)

Challenge:	Large language models store factual knowledge in parameters, but it can become outdated as the work evolves . pre-instruction-tuning improves ability of LLMs to absorb knowledge from new documents .
Approach:	They propose a method that instruction-tunes on questions prior to training on documents . they propose to use QA pairs to update factual knowledge of large language models .
Outcome:	The proposed method outperforms instruction-tuning on documents by 17.8%.

Copied to clipboard

Wenzheng Zhang, Xi Victoria Lin, Karl Stratos, Wen-tau Yih, Mingda Chen

On the Influence of Masking Policies in Intermediate Pre-training (2021.emnlp-main)

Challenge:	Retrieval-Augmented Generation (RAG) systems treat retrieval and generation as separate processes, requiring explicit textual queries to connect them.
Approach:	They propose a query-free RAG system that integrates retrieval and generation into a unified model.
Outcome:	The proposed system can achieve 3.6-11.5 accuracy improvements on unseen tasks . it allows models to express their information needs without human-specified queries .

Copied to clipboard

Qinyuan Ye, Belinda Z. Li, Sinong Wang, Benjamin Bolte, Hao Ma, Wen-tau Yih, Xiang Ren, Madian Khabsa

Altogether: Image Captioning via Re-aligning Alt-text (2024.emnlp-main)

Challenge:	Existing studies show that inserting an intermediate pre-training stage improves performance of masked language models.
Approach:	They propose methods to automate the discovery of optimal masking policies via direct supervision or meta-learning.
Outcome:	The proposed method outperforms the heuristic of masking named entities on TriviaQA and can be generalizable beyond that task.

Copied to clipboard

Hu Xu, Po-Yao Huang, Xiaoqing Tan, Ching-Feng Yeh, Jacob Kahn, Christine Jou, Gargi Ghosh, Omer Levy, Luke Zettlemoyer, Wen-tau Yih, Shang-Wen Li, Saining Xie, Christoph Feichtenhofer

Multimodality

Challenge:	Existing captioning models ignore existing alt-text metadata and lack transparency if training data is unknown.
Approach:	They propose an approach to edit and re-align alt-texts associated with images using human annotation.
Outcome:	The proposed approach improves image captions and improves text-to-image generation and zero-shot image classification tasks.

Reasoning about Actions and State Changes by Injecting Commonsense Knowledge (D18-1)

Copied to clipboard

Niket Tandon, Bhavana Dalvi, Joel Grus, Wen-tau Yih, Antoine Bosselut, Peter Clark

Reasoning

Challenge:	Recent work has shown impressive progress in comprehending procedural text, but their predictions can be inconsistent or highly improbable.
Approach:	They propose to incorporate global constraints and bias reading with corpora-based preferences to improve the predicted effects of actions in a paragraph.
Outcome:	The proposed model significantly outperforms earlier models on a benchmark dataset for procedural text comprehension (+8% relative gain) it avoids nonsensical predictions that earlier models make, and it is more robust than previous models.

Learning to Simulate Natural Language Feedback for Interactive Semantic Parsing (2023.acl-long)

Copied to clipboard

Hao Yan, Saurabh Srivastava, Yintao Tai, Sida I. Wang, Wen-tau Yih, Ziyu Yao

Semantic Parsing

Challenge:	Existing work on interactive semantic parsing relies on human annotations to train a model . prior work relied on human-annotated feedback data, which is prohibitively expensive and not scalable .
Approach:	They propose a task of simulating NL feedback for interactive semantic parsing . they propose evaluators to assess the quality of the simulated feedback .
Outcome:	The proposed simulator can generate high-quality NL feedback to boost the error correction ability of a specific parser.

DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrievers (2025.acl-long)

Copied to clipboard

Xueguang Ma, Xi Victoria Lin, Barlas Oguz, Jimmy Lin, Wen-tau Yih, Xilun Chen

Unsupervised Question Decomposition for Question Answering (2020.emnlp-main)

Challenge:	Large language models (LLMs) have shown strong effectiveness and robustness when fine-tuned as dense retrievers.
Approach:	They propose a training framework that leverages pruned LLMs to train smaller generalizable dense retrievers.
Outcome:	The proposed training framework offers better multilingual and long-context capabilities than traditional encoder-based retrievers and achieves strong performance across multiple tasks and languages.

Copied to clipboard

Ethan Perez, Patrick Lewis, Wen-tau Yih, Kyunghyun Cho, Douwe Kiela

Low-Resource NLP

Challenge:	Existing QA systems struggle to answer complex questions because information is scattered in different places.
Approach:	They propose an unsupervised algorithm that decomposes hard questions into simpler sub-questions . they propose an algorithm that can be used to generate a final answer from millions of questions .
Outcome:	The proposed algorithm decomposes hard questions into simpler sub-questions that existing QA systems can answer.

Model-based Interactive Semantic Parsing: A Unified Framework and A Text-to-SQL Case Study (D19-1)

Copied to clipboard

Ziyu Yao, Yu Su, Huan Sun, Wen-tau Yih

Programming Languages in NLP

Challenge:	Existing semantic parsing technologies are not well-suited for use in real-world applications.
Approach:	They propose a model-based intelligent agent that generates a clarification question in natural language . they propose 'interactive semantic parsing' with a human user in the loop .
Outcome:	The proposed approach improves both parsing accuracy and user confidence . it is demonstrated on two text-to-SQL datasets with different state-of-the-art parsers .

RoMQA: A Benchmark for Robust, Multi-evidence, Multi-answer Question Answering (2023.findings-emnlp)

Copied to clipboard

Victor Zhong, Weijia Shi, Wen-tau Yih, Luke Zettlemoyer

Responsible & Trustworthy NLP

Challenge:	Existing QA models are not robust to variations in question constraints, but can be made more robust by tuning on clusters of related questions.
Approach:	They introduce RoMQA, the first benchmark for robust, multi-evidence, multianswer question answering (QA) RoMQ contains clusters of related questions that are derived from the Wikidata knowledge graph .
Outcome:	The proposed model is the first benchmark for robust, multi-evidence, multianswer question answering (QA) compared to prior QA datasets, it has more human-written questions that require reasoning over more evidence text and have, on average, many more correct answers.

Everything Happens for a Reason: Discovering the Purpose of Actions in Procedural Text (D19-1)

Copied to clipboard

Bhavana Dalvi, Niket Tandon, Antoine Bosselut, Wen-tau Yih, Peter Clark

Responsible & Trustworthy NLP

Challenge:	XPAD is a new model that predicts actions' effects and their dependencies based on background knowledge . previous work on extracting sequences of actions from text has focused on identifying why they are the way they are .
Approach:	They propose a new model that biases effect predictions towards those that explain more of the actions in the paragraph and are more plausible with respect to background knowledge.
Outcome:	The proposed model outperforms existing systems on explaining actions by predicting dependencies while maintaining the performance on the original task in ProPara.

On Unifying Misinformation Detection (2021.naacl-main)

Copied to clipboard

Nayeon Lee, Belinda Z. Li, Sinong Wang, Pascale Fung, Hao Ma, Wen-tau Yih, Madian Khabsa

Responsible & Trustworthy NLP

Challenge:	On any given day, 2.5 quintillion bytes of information are created on the Internet, a figure that is only expected to increase in the coming years.
Approach:	They propose a general-purpose misinformation model that jointly models multiple domains of misinformation with a single, unified setup.
Outcome:	The proposed model is useful for few-shot learning of unseen misinformation tasks/datasets and generalizability to unseense events.

Reimagining Retrieval Augmented Language Models for Answering Queries (2023.findings-acl)

Copied to clipboard

Wang-Chiew Tan, Yuliang Li, Pedro Rodriguez, Richard James, Xi Victoria Lin, Alon Halevy, Wen-tau Yih

On the Efficacy of Adversarial Data Collection for Question Answering: Results from a Large-Scale Randomized Study (2021.acl-long)

Challenge:	Large language models (LLMs) are expensive to train, deploy, and maintain, both financially and in terms of environmental impact.
Approach:	They present a reality check on large language models and compare their predictions to retrieval-augmented language models.
Outcome:	The proposed models fare better on question answering tasks and have become the foundation of impressive demos like Chat-GPT.

Copied to clipboard

Divyansh Kaushik, Douwe Kiela, Zachary C. Lipton, Wen-tau Yih

Robustness in NLP

Challenge:	Existing studies have shown that adversarial data collection (ADC) models perform better on other adversarially collected data but are liable under plausible domain shifts.
Approach:	They conduct a large-scale controlled study on question answering by assigning workers at random to compose questions either adversarially (with a model in the loop) or in the standard fashion (without a modeling).
Outcome:	The proposed model performs better on other adversarial datasets but worse on diverse collection of out-of-domain evaluation sets.

How to Train Your Dragon: Diverse Augmentation Towards Generalizable Dense Retrieval (2023.findings-emnlp)

Copied to clipboard

Sheng-Chieh Lin, Akari Asai, Minghan Li, Barlas Oguz, Jimmy Lin, Yashar Mehdad, Wen-tau Yih, Xilun Chen