Papers by Tongshuang Wu
MoR: Better Handling Diverse Queries with a Mixture of Sparse, Dense, and Human Retrievers (2025.emnlp-main)
Copied to clipboard
| Challenge: | Different retrievers offer distinct, often complementary signals, but they are not optimal for all queries. |
| Approach: | They propose a zero-shot, weighted combination of heterogeneous retrievers . they validate this intuition by incorporating specialized non-oracle human information sources . |
| Outcome: | Experiments show that a mixture of heterogeneous retrievers outperforms each retriever and larger models by +10.8% and +3.9% on average. |
Fact-and-Reflection (FaR) Improves Confidence Calibration of Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on the confidence calibration of LLMs have not explored the effects of different prompting strategies on LLM performance. |
| Approach: | They propose Fact-and-Reflection prompting which improves LLM confidence calibration . they propose to use human cognition to elicit known "facts" and ask model to "reflect" over them . |
| Outcome: | The proposed method lowers the expected calibration error by 23.5% on multi-purpose QA tasks. |
It is AI’s Turn to Ask Humans a Question: Question-Answer Pair Generation for Children’s Story Books (2022.acl-long)
Copied to clipboard
| Challenge: | Existing question answering (QA) techniques are created mainly to answer questions asked by humans, but in educational applications, teachers often need to decide what questions to ask . |
| Approach: | They propose to use a fairytale-themed storybook as input to generate QA pairs that can test a student's comprehension skills. |
| Outcome: | The proposed system outperforms state-of-the-art QAG baseline systems and builds an interactive story-telling application for the future real-world deployment. |
DataFinder: Scientific Dataset Recommendation from Natural Language Descriptions (2023.acl-long)
Copied to clipboard
| Challenge: | Modern machine learning relies on datasets to develop and validate research ideas. |
| Approach: | They propose a dataset recommendation system that uses a training set and an evaluation set to help people find relevant datasets. |
| Outcome: | The proposed model finds more relevant search results than existing third-party search engines. |
Synthetic Multimodal Question Generation (2024.findings-emnlp)
Copied to clipboard
Ian Wu, Sravan Jayanthi, Vijay Viswanathan, Simon Rosenberg, Sina Pakazad, Tongshuang Wu, Graham Neubig
| Challenge: | Multimodal Retrieval Augmented Generation (MMRAG) is a powerful approach to question-answering over multimodal documents. |
| Approach: | They propose a synthetic data generation framework that leverages interplay between a retriever, large language model and large multimodal model to generate question and answer pairs directly from multimodal documents. |
| Outcome: | The proposed framework generates question and answer pairs from 1024 questions over Wikipedia documents and evaluates state-of-the-art models using it. |
Fantastic Questions and Where to Find Them: FairytaleQA – An Authentic Dataset for Narrative Comprehension (2022.acl-long)
Copied to clipboard
Ying Xu, Dakuo Wang, Mo Yu, Daniel Ritchie, Bingsheng Yao, Tongshuang Wu, Zheng Zhang, Toby Li, Nora Bradford, Branda Sun, Tran Hoang, Yisi Sang, Yufang Hou, Xiaojuan Ma, Diyi Yang, Nanyun Peng, Zhou Yu, Mark Warschauer
| Challenge: | Existing QA datasets rarely distinguish fine-grained reading skills, such as the understanding of varying narrative elements. |
| Approach: | They propose to use FairytaleQA to generate 10,580 questions based on 278 children-friendly stories to assess model's fine-grained learning skills. |
| Outcome: | The proposed dataset consists of 10,580 questions derived from 278 children-friendly stories, covering seven types of narrative elements or relations. |
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models (2021.acl-long)
Copied to clipboard
| Challenge: | Existing counterfactual generation methods rely on manual labor to create very few counterf actuals or only instantiate limited types of perturbations such as paraphrases or word substitutions. |
| Approach: | They propose a general-purpose counterfactual generator that allows for control over perturbation types and locations. |
| Outcome: | The proposed generator produces diverse sets of realistic counterfactuals that are useful in various applications. |
What Prompts Don’t Say: Understanding and Managing Underspecification in LLM Prompts (2026.findings-acl)
Copied to clipboard
| Challenge: | Under-specified prompts are 2x as likely to regress across model or prompt changes, authors show . eliot safina: a lack of explicit prompts can cause frustrations and failures . |
| Approach: | They propose requirements-aware prompt optimization mechanisms that improve performance by 4.8% over baselines. |
| Outcome: | The proposed mechanisms improve prompt performance by 4.8% over baselines. |
Errudite: Scalable, Reproducible, and Testable Error Analysis (P19-1)
Copied to clipboard
| Challenge: | Errudite is an interactive tool for informative error analysis. |
| Approach: | They propose an interactive tool for model and task agnostic error analysis . they propose a model-agnosed approach to model and tasks error analysis with Errudite . |
| Outcome: | The proposed tool enables reproducible error analyses with less effort and ambiguities in previous tools. |
Prompt2Model: Generating Deployable Models from Natural Language Instructions (2023.emnlp-demo)
Copied to clipboard
| Challenge: | Large language models (LLMs) are a step backward from traditional special-purpose NLP models . they require extensive computational resources for deployment and can be gated behind APIs . |
| Approach: | They propose a general-purpose method that takes a natural language task description and uses it to train a special-purpose model. |
| Outcome: | The proposed method outperforms a strong LLM by 20% while being 700 times smaller. |
Beyond Testers’ Biases: Guiding Model Testing with Knowledge Bases using LLMs (2023.findings-emnlp)
Copied to clipboard
Chenyang Yang, Rishabh Rustogi, Rachel Brower-Sinning, Grace Lewis, Christian Kaestner, Tongshuang Wu
| Challenge: | Identifying what to test is a step that is largely ignored and poorly supported. |
| Approach: | They propose an interactive tool that supports requirements elicitation for guiding model testing. |
| Outcome: | The proposed tool can help practitioners test models in real-world settings . |
Better Synthetic Data by Retrieving and Transforming Existing Datasets (2024.findings-acl)
Copied to clipboard
| Challenge: | despite advances in large language models, task-specific data is not available for many use cases . a new method to improve automated dataset generation uses publicly available datasets . |
| Approach: | They propose a method to make better use of existing datasets to improve automatic dataset generation. |
| Outcome: | The proposed method outperforms existing methods on language-based tasks . it significantly increases diversity and difficulty of generated data on many tasks compared to other methods . |
SPHERE: An Evaluation Card for Human-AI Systems (2025.findings-acl)
Copied to clipboard
Dora Zhao, Qianou Ma, Xinran Zhao, Chenglei Si, Chenyang Yang, Ryan Louie, Ehud Reiter, Diyi Yang, Tongshuang Wu
| Challenge: | Existing evaluation methods and standards for human-AI systems are unclear, especially for large language models. |
| Approach: | They propose an evaluation card SPHERE which provides a template for evaluation protocols . they outline current evaluation practices and areas for improvement . |
| Outcome: | The evaluation card provides a template for designing evaluation protocols . it outlines current evaluation practices and areas for improvement . |
Large Language Models Help Humans Verify Truthfulness – Except When They Are Convincingly Wrong (2024.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly used for accessing information on the web. |
| Approach: | They conduct experiments with 80 crowdworkers to compare LLMs with search engines . they ask LLM to provide contrastive information to reduce over-reliance on LLM . |
| Outcome: | The results show that LLMs can outperform search engines but not LLM explanations . the study shows that LMS explanations are not reliable replacements for reading retrieved passages compared to search engines alone. |
Are Shortest Rationales the Best Explanations for Human Understanding? (2022.acl-short)
Copied to clipboard
| Challenge: | Existing models favor extracting the shortest possible rationales to explain model predictions . however, this assumption has yet to be validated . |
| Approach: | They propose a model that extracts rationales at any target length from text inputs . they show that rationale lengths too short do not help humans predict labels better . |
| Outcome: | The proposed model achieves compatible end-task performance and human-annotated rationale agreement compared to baseline models . |
NewsSense: Reference-free Verification via Cross-document Comparison (2023.emnlp-demo)
Copied to clipboard
| Challenge: | Various solutions have been proposed to assist with news reading, including media bias charts, which treat each news source as a whole, without digging into specific articles or topics. |
| Approach: | They propose a novel sensemaking tool to integrate news articles on a central topic. |
| Outcome: | The novel reading interface can help users identify key information, verify the credibility of news articles, explore different perspectives, and understand what content is supported, contradicted, or missing. |
Tailor: Generating and Perturbing Text with Semantic Controls (2022.acl-long)
Copied to clipboard
| Challenge: | Existing studies train task-specific generators, relying on training a model for every perturbation. |
| Approach: | They propose a semantically-controlled text generation system that modifies sentences to match target attributes. |
| Outcome: | The proposed system produces textual outputs conditioned on control codes derived from semantic representations. |
RECAP: An End-to-End Platform for Capturing, Replaying, and Analyzing AI-Assisted Programming Interactions (2026.acl-demo)
Copied to clipboard
| Challenge: | Deployed in a university software engineering course, RECAP captured 2,034 prompts and 8,239 code edits from 41 students across a multi-week project. |
| Approach: | They propose an open-source platform that passively records AI chat sessions and fine-grained code edits inside VS Code without disrupting the developer’s workflow. |
| Outcome: | The open-source platform captures 2,034 prompts and 8,239 code edits from 41 students across a multi-week project. |
BiasX: “Thinking Slow” in Toxic Content Moderation with Explanations of Implied Social Biases (2023.emnlp-main)
Copied to clipboard
| Challenge: | Toxicity annotators and content moderators often default to mental shortcuts when making decisions, leading to subtle toxicity being missed and seemingly harmless content being over-detected. |
| Approach: | They propose a framework that provides AI-generated explanations of statements’ implied social biases to enhance content moderation setups. |
| Outcome: | The proposed framework significantly improves content moderation setups by enabling users to think more thoroughly about their decisions. |
Designing, Evaluating, and Learning from Humans Interacting with NLP Models (2023.emnlp-tutorial)
Copied to clipboard
| Challenge: | This tutorial will cover how to conduct human-in-the-loop usability evaluations to ensure that models are capable of interacting with humans. |
| Approach: | They will provide a systematic overview of key considerations and effective approaches for studying human-NLP model interactions. |
| Outcome: | This tutorial will cover how to conduct human-in-the-loop usability evaluations to ensure that models are capable of interacting with humans. |
cAST: Enhancing Code Retrieval-Augmented Generation with Structural Chunking via Abstract Syntax Tree (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing line-based chunking heuristics often break semantic structures, splitting functions or merging unrelated code. |
| Approach: | They propose a structure-aware method that breaks large AST nodes into smaller chunks . this method generates self-contained, semantically coherent units across programming languages . |
| Outcome: | The proposed method boosts Recall@5 by 4.3 points on RepoEval retrieval and Pass@1 by 2.67 points on SWE-bench generation. |
Scaling Collaborative Effort with Agents (2026.findings-acl)
Copied to clipboard
Shannon Zejiang Shen, Valerie Chen, Ken Gu, Alexis Ross, Zixian Ma, Jillian Ross, Alex Gu, Chenglei Si, Wayne Chi, Andi Peng, Jocelyn J Shen, Ameet Talwalkar, Tongshuang Wu, David Sontag
| Challenge: | Current evaluations of agents focus on producing high-quality, final outputs in one shot, failing to account for the inherently iterative nature of many real-world problems. |
| Approach: | They propose a framework that captures how an agent’s utility grows with increasing user involvement. |
| Outcome: | The proposed framework captures how an agent’s utility grows with increasing user involvement, revealing a missing ingredient in agent design: the ability to sustain engagement and scaffold user understanding. |
Beyond Accuracy: Behavioral Testing of NLP Models with CheckList (2020.acl-main)
Copied to clipboard
| Challenge: | In a recent study, we show that holding-out data can overestimate performance of NLP models. |
| Approach: | They propose a task-agnostic methodology for testing NLP models using a matrix of general linguistic capabilities and test types that facilitate comprehensive test ideation. |
| Outcome: | The proposed method identifies critical failures in commercial and state-of-the-art models. |