Papers by Eunsol Choi

CaLMQA: Exploring culturally specific long-form question answering across 23 languages (2025.acl-long)

Challenge:	Temporal misalignment is a problem for knowledge-intensive tasks where models must rely on data from the past to make predictions.
Approach:	They propose a temporal misalignment task to predict how long a given fact will remain true.
Outcome:	The proposed task improves calibration for knowledge-intensive tasks under temporal misalignment by discarding volatile facts.

Copied to clipboard

Shane Arora, Marzena Karpinska, Hung-Ting Chen, Ipsita Bhattacharjee, Mohit Iyyer, Eunsol Choi

Multilinguality

Challenge:	Despite rising global usage of large language models, their ability to generate long-form answers to culturally specific questions remains unexplored in many languages.
Approach:	They perform the first study of textual multilingual long-form QA by creating a dataset of culturally specific questions across 23 different languages.
Outcome:	The results show that the best models make critical surface-level errors for many languages and their understanding of diverse cultures.

Ultra-Fine Entity Typing (P18-1)

Copied to clipboard

Eunsol Choi, Omer Levy, Yejin Choi, Luke Zettlemoyer

Information Extraction & Text Mining

Challenge:	Experimental results show that a model that can predict ultra-fine types can be crowd-sourced . head words indicate the type of the noun phrases they appear in, and are important for context-sensitive tasks .
Approach:	They propose a task where sentences are given with an entity mention . they introduce a new type of distant supervision: head words, which indicate the type of noun phrases they appear in.
Outcome:	The proposed model can predict ultra-fine types at varying granularity and performs well on a fine-grained entity typing benchmark.

SAGE: Steerable Agentic Data Generation for Deep Search with Execution Feedback (2026.findings-eacl)

Copied to clipboard

Fangyuan Xu, Rujun Han, Yanfei Chen, Zifeng Wang, I-Hung Hsu, Jun Yan, Vishy Tirumalashetty, Eunsol Choi, Tomas Pfister, Chen-Yu Lee

Reasoning

Challenge:	High-quality, complex question-answer pairs are pivotal for training and evaluating capable deep search agents.
Approach:	They propose a pipeline that generates high-quality, difficulty-controlled deep search question-answer pairs for a given corpus and a target difficulty level.
Outcome:	The proposed pipeline generates high-quality, difficulty-controlled deep search question-answer pairs for a given corpus and a target difficulty level.

DiffQG: Generating Questions to Summarize Factual Changes (2023.eacl-main)

Copied to clipboard

Jeremy R. Cole, Palak Jain, Julian Martin Eisenschlos, Michael J.Q. Zhang, Eunsol Choi, Bhuwan Dhingra

Text Generation

Challenge:	Existing methods to identify factual changes between paired documents are limited . specialized entailment-like resources and models have been applied to fact verification .
Approach:	They propose to represent factual changes between paired documents as question-answer pairs . they propose to generate a discriminating question given an answer span such that the question is answerable by one passage but not the other .
Outcome:	The proposed model can flexibly and concisely capture the updated contents of paired documents.

RefreshKV: Updating Small KV Cache During Long-form Generation (2025.acl-long)

Copied to clipboard

Fangyuan Xu, Tanya Goyal, Eunsol Choi

Challenge:	Existing methods for generating long sequences of tokens are expensive and require memory and computation resources.
Approach:	They propose a method that alternates between full context attention and attention over a subset of input tokens during generation.
Outcome:	The proposed method achieves comparable speedup to eviction-based methods while improving performance for various long-form generation tasks.

Misinfo Reaction Frames: Reasoning about Readers’ Reactions to News Headlines (2022.acl-long)

Copied to clipboard

Saadia Gabriel, Skyler Hallinan, Maarten Sap, Pemi Nguyen, Franziska Roesner, Eunsol Choi, Yejin Choi

Reasoning

Challenge:	Empirical results confirm that it is indeed possible for neural models to predict the prominent patterns of readers’ reactions to previously unseen news headlines.
Approach:	They propose a pragmatic formalism for modeling how readers might react to a news headline . they propose 'misinfo' frames, which can be used to model reader perceptions of news reliability .
Outcome:	The proposed model can predict readers' reactions to previously unseen headlines.

Complex Claim Verification with Evidence Retrieved in the Wild (2024.naacl-long)

Copied to clipboard

Jifan Chen, Grace Kim, Aniruddh Sriram, Greg Durrett, Eunsol Choi

Fact & Claim Verification

Challenge:	Prior work makes simplifying assumptions in retrieval that depart from real-world use cases: no access to evidence, access to curated evidence, or access to published evidence after a claim was made.
Approach:	They propose a pipeline to check claims using raw evidence from the web . they restrict their retriever to only search documents available prior to the claim's making .
Outcome:	The proposed method is based on a political claim dataset and shows that the evidence summary produced by the system is reliable and relevant to answering key questions.

Clarify When Necessary: Resolving Ambiguity Through Interaction with LMs (2025.findings-naacl)

Copied to clipboard

Michael JQ Zhang, Eunsol Choi

Semantic Text Processing

Challenge:	Ambiguity is embedded throughout natural language, and even simple utterances can have multiple interpretations when read in isolation.
Approach:	They propose a task-agnostic framework for evaluating a system’s ability to determine when to ask for clarification.
Outcome:	The proposed framework outperforms existing uncertainty estimation approaches at identifying predictions that will benefit from clarification.

pair2vec: Compositional Word-Pair Embeddings for Cross-Sentence Inference (N19-1)

Copied to clipboard

Mandar Joshi, Eunsol Choi, Omer Levy, Daniel Weld, Luke Zettlemoyer

Representation Learning

Challenge:	Existing inference models that rely heavily on unsupervised single-word embeddings struggle to learn implied relationships between pairs of words.
Approach:	They propose to use word embeddings to learn and use background knowledge about implied relationships between words that are crucial for cross-sentence inference problems.
Outcome:	The proposed models gain 2.7% on the recently released SQuAD 2.0 and 1.3% on MultiNLI, and 8.8% on the adversarial SQu AD datasets.

From Distributional to Overton Pluralism: Investigating Large Language Model Alignment (2025.naacl-long)

Copied to clipboard

Thom Lake, Eunsol Choi, Greg Durrett

How Do We Answer Complex Questions: Discourse Structure of Long-form Answers (2022.acl-long)

Challenge:	a large language model's (LLM) output distribution is changed by an alignment process . a recent study shows that aligned models surface information that cannot be recovered from base models without fine-tuning.
Approach:	They analyze two aspects of the alignment process that change output distributions . they find alignment suppresses irrelevant and unhelpful content .
Outcome:	The proposed model can be imitated without fine-tuning by using in-context examples and lower-resolution semantic hints about response content.

Copied to clipboard

Fangyuan Xu, Junyi Jessy Li, Eunsol Choi

Challenge:	Recent work explored long-form answers, where answers are free-form texts consisting of multiple sentences.
Approach:	They develop an ontology of six sentence-level functional roles for long-form answers . they annotate 3.9k sentences in 640 answer paragraphs and train a strong classifier .
Outcome:	The proposed model-generated answers agree less with model-driven answers than human-written answers.

Simulating Bandit Learning from User Feedback for Extractive Question Answering (2022.acl-long)

Copied to clipboard

Ge Gao, Eunsol Choi, Yoav Artzi

Decontextualization: Making Sentences Stand-Alone (2021.tacl-1)

Challenge:	Explicit feedback from users can be used to continually improve system performance.
Approach:	They study the potential of learning from user feedback for extractive question answering by simulating feedback using supervised data.
Outcome:	The proposed model improves on a few examples and can be deployed in new domains without any data annotation effort.

Copied to clipboard

Eunsol Choi, Jennimaria Palomaki, Matthew Lamm, Tom Kwiatkowski, Dipanjan Das, Michael Collins

Explainability & Interpretability in NLP

Challenge:	Taking excerpts of text can be problematic, as key pieces may not be explicit in a local window.
Approach:	They define a problem of sentence decontextualization by rewriting a sentence to be interpretable out of context while preserving its meaning.
Outcome:	The proposed method can be used in question answering and document understanding tasks.

TyDiP: A Dataset for Politeness Classification in Nine Typologically Diverse Languages (2022.findings-emnlp)

Copied to clipboard

Anirudh Srinivasan, Eunsol Choi

Generating Literal and Implied Subquestions to Fact-check Complex Claims (2022.emnlp-main)

Challenge:	Whether politeness phenomena and strategies are universal across languages or not have been controversial among sociologists and linguists.
Approach:	They create a dataset containing three-way politeness annotations for 500 examples in each language, totaling 4.5K examples.
Outcome:	The proposed model shows a robust zero-shot transfer ability, but falls short of estimated human accuracy significantly.

Copied to clipboard

Jifan Chen, Aniruddh Sriram, Eunsol Choi, Greg Durrett

Reasoning

Challenge:	Existing fact-checking systems are not reliable because it is unclear which parts of a claim are true and which are not.
Approach:	They propose to decompose a political claim into a comprehensive set of yes-no subquestions whose answers influence the veracity of the claim.
Outcome:	The proposed models can decompose a complex claim into a comprehensive set of yes-no subquestions whose answers influence the veracity of the claim.

Crafting In-context Examples according to LMs’ Parametric Knowledge (2024.findings-naacl)

Copied to clipboard

Yoonsang Lee, Pranav Atreya, Xi Ye, Eunsol Choi

MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension (D19-58)

Challenge:	In-context examples can improve the performance of knowledge-rich tasks such as question answering by triggering a language model to surface information stored in its parametric knowledge.
Approach:	They propose to construct in-context example sets based on model's parametric knowledge by prompting models with 'unknown' examples.
Outcome:	The proposed model can perform better on in-context examples in three multi-answer question answering datasets, and prompting with ‘unknown’ examples decreases the performance.

Copied to clipboard

Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, Danqi Chen

Improving LLM-as-a-Judge Inference with the Judgment Distribution (2025.findings-emnlp)

Challenge:	MRQA datasets have been used to benchmark progress in general-purpose language understanding.
Approach:	They propose to combine 18 question answering datasets into one shared task to evaluate their generalization capabilities.
Outcome:	The best system achieved an average F1 score of 72.5 on the 12 held-out datasets, 10.7 absolute points higher than baseline based on BERT.

Copied to clipboard

Victor Wang, Michael JQ Zhang, Eunsol Choi

Challenge:	Using language models to scalably approximate human preferences on text quality (LLM-as-a-judge) is a standard practice applicable to many tasks.
Approach:	They propose to use LLM judges to approximate human preferences on text quality by using distributional output over judgment tokens.
Outcome:	The proposed method outperforms taking the mode (i.e. greedy decoding) in all evaluation settings, and incorporating risk aversion improves performance.

Continually Improving Extractive QA via Human Feedback (2023.emnlp-main)

Copied to clipboard

Ge Gao, Hung-Ting Chen, Yoav Artzi, Eunsol Choi

Learning with Different Amounts of Annotation: From Zero to Many Labels (2021.emnlp-main)

Challenge:	a study of extractive question answering systems using human feedback shows promising potential for continual learning.
Approach:	They study extractive question answering system by using user feedback to improve it . they design and deploy an iterative approach where users ask questions and provide feedback .
Outcome:	The proposed model improves over time across different data regimes and domains . human user feedback is more affordable and abundant than annotations provided by trained experts .

Copied to clipboard

Shujian Zhang, Chengyue Gong, Eunsol Choi

Challenge:	a lack of annotator agreement can hinder training of NLP systems . we propose a learning algorithm that can learn from training examples with zero, one, or multiple labels.
Approach:	They propose an annotation distribution scheme that assigns multiple labels to training examples . they propose a learning algorithm that can learn from training examples with different amount of annotation .
Outcome:	The proposed method achieves consistent gains in two tasks, suggesting distributing labels unevenly among training examples can be beneficial for many NLP tasks.

User Feedback in Human-LLM Dialogues: A Lens to Understand Users But Noisy as a Learning Signal (2025.emnlp-main)

Copied to clipboard

Yuhan Liu, Michael JQ Zhang, Eunsol Choi

QED: A Framework and Dataset for Explanations in Question Answering (2021.tacl-1)

Challenge:	a recent study shows that asking for direct user feedback can be disruptive . we examine whether incorporating the contents of user feedback improves model performance .
Approach:	They analyze user feedback in the user-LLM conversation logs and harvest learning signals from it.
Outcome:	The proposed approach can lead to model degradation on two user-LM interaction datasets.

Copied to clipboard

Matthew Lamm, Jennimaria Palomaki, Chris Alberti, Daniel Andor, Eunsol Choi, Livio Baldini Soares, Michael Collins

Exploring Design Choices for Building Language-Specific LLMs (2024.findings-emnlp)

Challenge:	Existing question answering systems provide no explanation of reasoning that leads to answer . linguistically informed, extensible framework provides explanations in question answering .
Approach:	They propose a linguistically informed, extensible framework for explanations in question answering . they propose an expert-annotated dataset of QED explanations built upon a subset of the Natural Questions dataset .
Outcome:	The proposed framework improves the ability of untrained raters to spot errors in QA datasets.

Copied to clipboard

Atula Tejaswi, Nilesh Gupta, Eunsol Choi

TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages (2020.tacl-1)

Challenge:	Prior work focused on building multilingual models that cover a broad spectrum of languages.
Approach:	They conduct systematic experiments on how design choices impact the adapted LLM, both in terms of efficiency and end task performance.
Outcome:	The proposed model performs better on English-centric models than multilingual models despite poor performance on low-resource languages.

Copied to clipboard

Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, Jennimaria Palomaki

SituatedQA: Incorporating Extra-Linguistic Contexts into QA (2021.emnlp-main)

Challenge:	Existing models for multilingual modeling are based on a set of typological features that are used to express meaning in languages such as English.
Approach:	They present a question-answer-typed question-referenced dataset that covers 11 typologically diverse languages with 204K question-and-answered pairs.
Outcome:	The proposed dataset covers 11 typologically diverse languages with 204K question-answer pairs.

Copied to clipboard

Michael Zhang, Eunsol Choi

QuAC: Question Answering in Context (D18-1)

Challenge:	Existing models struggle with producing answers that are frequently updated or from uncommon locations.
Approach:	They propose an open-retrieval QA dataset where systems must produce the correct answer given the context.
Outcome:	The proposed dataset shows that existing models struggle with producing answers that are frequently updated or from uncommon locations.

Copied to clipboard

Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, Luke Zettlemoyer

Textless Speech-to-Speech Translation With Limited Parallel Data (2024.findings-emnlp)

Challenge:	a dataset for Question Answering in Context contains 14K information-seeking QA dialogs . questions are often more open-ended, unanswerable, or only meaningful within the dialog context .
Approach:	They propose a dataset for Question Answering in Context that contains 14K dialogs . they use a student to ask questions about a Wikipedia section and a teacher to answer them .
Outcome:	The proposed dataset underperforms humans in a number of reference models . the dataset contains 14K information-seeking dialogs over sections from Wikipedia .

Copied to clipboard

Anuj Diwan, Anirudh Srinivasan, David Harwath, Eunsol Choi

Multimodality

Challenge:	Existing speech-to-speech translation models either leverage text as an intermediate step or require hundreds of hours of parallel speech data.
Approach:	They propose a framework for training textless S2ST models that require dozens of hours of parallel speech data.
Outcome:	The proposed model achieves reasonable performance on three domains with single-speaker synthesized speech.

Concise Answers to Complex Questions: Summarization of Long-form Answers (2023.acl-long)

Copied to clipboard

Abhilash Potluri, Fangyuan Xu, Eunsol Choi

Information Extraction & Text Mining

Challenge:	Long-form question answering systems provide rich information by presenting paragraph-level answers, but not all information is required to answer the question.
Approach:	They propose an extract-and-decontextualize approach to summarize long-form answers using state-of-the-art models.
Outcome:	The proposed extract-and-decontextualize approach improves the quality of the extractive summary, exemplifying its potential in the summarization task.

Quantifying Train-Evaluation Overlap with Nearest Neighbors (2023.findings-acl)

Copied to clipboard

Gauri Kambhatla, Thuy Nguyen, Eunsol Choi

Challenge:	Using a novel metric, we quantify train-evaluation overlap in benchmark datasets to evaluate model generalization.
Approach:	They quantify train-evaluation overlap as a measure of an individual dataset’s adequacy to evaluate model generalization over a wide range of datasets.
Outcome:	The proposed metric quantifies train-evaluation overlap, providing insights for constructing datasets to study generalization.

Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidence (2022.emnlp-main)

Copied to clipboard

Hung-Ting Chen, Michael Zhang, Eunsol Choi

XOR QA: Cross-lingual Open-Retrieval Question Answering (2021.naacl-main)

Challenge:	Existing work on question answering models relies on retrieved documents for provenance, but recent studies show that models can retain vast amounts of factual knowledge . retrieval-based generation approaches combine parametric knowledge sources with a large number of retrieved evidence documents, achieving state-of-the-art performance on open retrieval datasets.
Approach:	They propose to use parametric and parametric knowledge to generate free-form questions from retrieved evidence documents.
Outcome:	The proposed model can use parametric and parametric knowledge to generate free-form answers from retrieved evidence documents.

Copied to clipboard

Akari Asai, Jungo Kasai, Jonathan Clark, Kenton Lee, Eunsol Choi, Hannaneh Hajishirzi

Language Models Don’t Know What You Want: Evaluating Personalization in Deep Research Needs Real Users (2026.acl-long)

Challenge:	a dataset of 40k information-seeking questions across seven languages is used to answer multilingual question answering tasks.
Approach:	They propose a task framework that allows questions from one language to be answered via answer content from another language.
Outcome:	The proposed framework can be used to answer questions from one language to another . the dataset was built on 40K questions across 7 languages, but could not find same-language answers .

Copied to clipboard

Nishant Balepur, Malachi Hamada, Varsha Kishore, Sergey Feldman, Amanpreet Singh, Pao Siangliulue, Joseph Chee Chang, Eunsol Choi, Jordan Lee Boyd-Graber, Aakanksha Naik

Semantic Text Processing

Challenge:	Earlier research used real users to push personalization, but easy-to-use judges have been criticized for not adopting online studies.
Approach:	They propose a personalized action-following tool that infers a user's research interests and proposes personalized actions for a query.
Outcome:	The proposed tool beats baselines in citation metrics and personalized action-following with an online version of MySQA.

Entities as Experts: Sparse Memory Access with Entity Supervision (2020.emnlp-main)

Copied to clipboard

Thibault Févry, Livio Baldini Soares, Nicholas FitzGerald, Eunsol Choi, Tom Kwiatkowski

Semantic Text Processing

Challenge:	Unlike previous attempts to integrate entity knowledge into sequence models, EaE’s entity representations are learned directly from text.
Approach:	They propose a model that can access distinct memories of entities mentioned in a piece of text and a new architecture that can do this.
Outcome:	The proposed model outperforms an encoder-generator Transformer model with 10x the parameters on a task that requires 10x more parameters to answer.

Scaling Rich Style-Prompted Text-to-Speech Datasets (2025.emnlp-main)

Copied to clipboard

Anuj Diwan, Zhisheng Zheng, David Harwath, Eunsol Choi

Multimodality

Challenge:	Existing datasets that only cover basic tags are limited in their scale or coverage of style tags.
Approach:	They propose a large-scale dataset that annotates speech utterances with rich style captions.
Outcome:	The proposed dataset scales speech utterances with rich style captions for the first time.

When to Use Efficient Self Attention? Profiling Text, Speech and Image Transformer Variants (2023.acl-short)

Copied to clipboard

Anuj Diwan, Eunsol Choi, David Harwath

Multimodality

Challenge:	Existing models focus on improving the efficiency of self-attention, but in practice they may be slower, especially given modest input lengths that are typical of many tasks.
Approach:	They propose a novel local-attention variant of a self-supervised speech model that uses input length thresholds to identify bottlenecks.
Outcome:	The proposed model is based on a self-attention-based model with a high input length threshold.

Open-World Evaluation for Retrieving Diverse Perspectives (2025.naacl-long)

Copied to clipboard

Hung-Ting Chen, Eunsol Choi

Can NLI Models Verify QA Systems’ Predictions? (2021.findings-emnlp)

Challenge:	Existing retrieval systems only cover diverse perspectives on 33.74% of the examples . existing systems only focus on relevance to the question, ignoring diversity.
Approach:	They build a Benchmark for Retrieval Diversity for Subjective questions (BERDS) based on a question and diverse perspectives associated with the question . they evaluate retrievers paired with a corpus to determine whether each document contains a perspective .
Outcome:	The proposed approach improves retrieval diversity on complex questions . existing retrieval systems only cover diverse perspectives on 33.74% of the examples .

Copied to clipboard

Jifan Chen, Eunsol Choi, Greg Durrett

Challenges in Information-Seeking QA: Unanswerable Questions and Paragraph Retrieval (2021.acl-long)

Challenge:	Recent question answering systems perform well on benchmark datasets, but are not always well-calibrated to spot spurious answers under distribution shifts.
Approach:	They propose to use natural language inference to verify whether answers are correct . they leverage large pre-trained models and recent prior datasets to construct powerful question conversion and decontextualization modules.
Outcome:	The proposed approach improves the confidence estimation of a QA model across different domains, evaluated in a selective QA setting.

Copied to clipboard

Akari Asai, Eunsol Choi

Neural Metaphor Detection in Context (D18-1)

Challenge:	Existing pretrained language models have solved reading comprehension benchmarks, but datasets with information-seeking queries remain challenging.
Approach:	They analyze why answering information-seeking queries is more challenging . they manually annotate 800 unanswerable examples across six languages .
Outcome:	The proposed model outperforms human annotators on 800 unanswerable examples across six languages.

Copied to clipboard

Ge Gao, Eunsol Choi, Yejin Choi, Luke Zettlemoyer

Sentiment Analysis

Challenge:	Existing models focus on limited forms of linguistic context, such as unigrams.
Approach:	They propose end-to-end neural models for detecting metaphorical word use in context . they show that bi-directional biLSTM models which operate on complete sentences work well .
Outcome:	The proposed models show that they can learn rich contextual word representations . they are compared to previous models which focused on limited linguistic context .

Beyond Counting Datasets: A Survey of Multilingual Dataset Construction and Necessary Resources (2022.findings-emnlp)

Copied to clipboard

Xinyan Yu, Trina Chatterjee, Akari Asai, Junjie Hu, Eunsol Choi

Multilinguality

Challenge:	Existing studies have examined the quality of labeled data in non-English languages.
Approach:	They annotate how datasets are created, input text and label sources, tools used to build them and what they study.
Outcome:	The results show that language-proficient NLP researchers' estimated availability correlates with dataset availability.

Entity Cloze By Date: What LMs Know About Unseen Entities (2022.findings-naacl)

Copied to clipboard

Yasumasa Onoe, Michael Zhang, Eunsol Choi, Greg Durrett

A Critical Evaluation of Evaluations for Long-form Question Answering (2023.acl-long)

Challenge:	Existing literature provides benchmarks to measure LMs' knowledge about entities .
Approach:	They propose a framework to analyze what language models can infer about new entities that did not exist when they were pretrained.
Outcome:	The proposed framework shows that models more informed about the entities achieve lower perplexity on this benchmark.

Copied to clipboard

Fangyuan Xu, Yixiao Song, Mohit Iyyer, Eunsol Choi

Knowing More About Questions Can Help: Improving Calibration in Question Answering (2021.findings-acl)

Challenge:	Long-form question answering (LFQA) is an emerging research area within QA . however, its flexibility poses enormous challenges for evaluation .
Approach:	They conduct the first targeted study of the evaluation of long-form answers, covering both human and automatic evaluation practices.
Outcome:	The proposed evaluations cover human and automatic evaluations.

Copied to clipboard

Shujian Zhang, Chengyue Gong, Eunsol Choi

Can LMs Learn New Entities from Descriptions? Challenges in Propagating Injected Knowledge (2023.acl-long)

Challenge:	Existing work on calibration focuses on model confidence, such as the max probability of the predicted class.
Approach:	They propose a calibration method which estimates whether model correctly predicts answer for each question.
Outcome:	The proposed calibration method achieves 5-10% gains on reading comprehension benchmarks.

Copied to clipboard

Yasumasa Onoe, Michael Zhang, Shankar Padmanabhan, Greg Durrett, Eunsol Choi

KIWI: A Dataset of Knowledge-Intensive Writing Instructions for Answering Research Questions (2024.findings-acl)

Challenge:	Existing methods for updating knowledge show little propagation of injected knowledge.
Approach:	They propose to inject individual facts into LMs and evaluate whether they can propagate injected facts while not changing predictions on other contexts.
Outcome:	The proposed model can make inferences based on injected facts and propagate them . existing methods show little propagation of injected knowledge .

Copied to clipboard

Fangyuan Xu, Kyle Lo, Luca Soldaini, Bailey Kuehl, Eunsol Choi, David Wadden

BenchMarker: An Education-Inspired Toolkit for Highlighting Flaws in Multiple-Choice Benchmarks (2026.acl-long)

Challenge:	Large language models (LLMs) are increasingly used as conversational agents.
Approach:	They construct a dataset of knowledge-intensive writing instructions to evaluate LLMs' ability to follow user instructions.
Outcome:	The proposed model fails to integrate new information into an existing answer and perform precise and unambiguous edits.

Copied to clipboard

Nishant Balepur, Bhavya Rajasekaran, Hyunjin Jane Oh, Michael Xie, Atrey Desai, Vipul Gupta, Steven James Moore, Eunsol Choi, Rachel Rudinger, Jordan Lee Boyd-Graber

Challenge:	Multiple-choice question answering (MCQ) is standard in NLP, but benchmarks lack rigorous quality control.
Approach:	They propose an education-inspired toolkit that uses LLM judges to flag flaws in MCQs . they validate the tool with annotations and run it to audit 12 benchmarks based on 19-rule education rubric .
Outcome:	The proposed toolkit flags three common MCQ flaws based on a 19-rule education rubric . contaminated MCqs tend to inflate accuracy, while writing errors lower it and change rankings .

Modeling Exemplification in Long-form Question Answering via Retrieval (2022.naacl-main)

Copied to clipboard

Shufan Wang, Fangyuan Xu, Laure Thompson, Eunsol Choi, Mohit Iyyer