Papers by Tushar Khot
Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies (2021.tacl-1)
Copied to clipboard
| Challenge: | Existing questions that explicitly describe the process for deriving the answer are often implicit. |
| Approach: | They propose a question answering benchmark where the required reasoning steps are implicit in the question and should be inferred using a strategy. |
| Outcome: | The proposed model is short, topic-diverse, and covers a wide range of strategies. |
MoNaCo: More Natural and Complex Questions for Reasoning Across Dozens of Documents (2026.tacl-1)
Copied to clipboard
Tomer Wolfson, Harsh Trivedi, Mor Geva, Yoav Goldberg, Dan Roth, Tushar Khot, Ashish Sabharwal, Reut Tsarfaty
| Challenge: | Automated agents powered by large language models are becoming more ingrained into how people seek information . but evaluation benchmarks for LLMs rarely feature natural questions that are time-consuming . a new benchmark, MoNaCo, aims to address this gap by eliciting and manually answering time-wasting questions . |
| Approach: | They propose a benchmark of 1,315 natural and time-consuming questions that require dozens of intermediate steps to solve. |
| Outcome: | MoNaCo benchmarks achieve at least 61.2% F1 in real-world time-consuming questions hampered by low recall and hallucinations . Frontier LLMs evaluated on MoN achieving at least 61% F1, harmed by low memory and halluzinations. |
UNIFIEDQA: Crossing Format Boundaries with a Single QA System (2020.findings-emnlp)
Copied to clipboard
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, Hannaneh Hajishirzi
| Challenge: | Question answering (QA) tasks have been posed using a variety of formats . a new study aims to develop specialized QA models that can be used to train QA systems . |
| Approach: | They build a pre-trained question answering model that performs well across 19 QA datasets . they argue that format-specialized models can limit the ability to teach reasoning . |
| Outcome: | a new model that trains on QA datasets performs on par with 8 models trained on individual datasets . a single model that trained on UNIFIEDQA performs well on 19 QA data . |
IIRC: A Dataset of Incomplete Information Reading Comprehension Questions (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing reading comprehension tasks focus on questions for which the contexts provide all the information required to answer them, thus not evaluating a system’s performance at identifying a potential lack of sufficient information and locating sources for that information. |
| Approach: | They propose to use a dataset with 13K questions over paragraphs from English Wikipedia that provide only partial information to answer them, with the missing information occurring in one or more linked documents. |
| Outcome: | The proposed model achieves 31.1% F1 on the reading comprehension task, while estimated human performance is 88.4%. |
ADaPT: As-Needed Decomposition and Planning with Language Models (2024.findings-naacl)
Copied to clipboard
Archiki Prasad, Alexander Koller, Mareike Hartmann, Peter Clark, Ashish Sabharwal, Mohit Bansal, Tushar Khot
| Challenge: | Large Language Models (LLMs) are increasingly being used for interactive decision-making tasks requiring planning and adapting to the environment. |
| Approach: | They propose an approach that explicitly plans and decomposes complex sub-tasks when the LLM is unable to execute them. |
| Outcome: | The proposed approach significantly outperforms established strong baselines, achieving success rates up to 28.3% higher in ALFWorld, 27% in WebShop, and 33% in TextCraft. |
Temporal Reasoning on Implicit Events from Distant Supervision (2021.naacl-main)
Copied to clipboard
| Challenge: | a novel temporal reasoning dataset evaluates the degree to which systems understand implicit events . state-of-the-art models struggle when predicting temporal relationships between implicit and explicit events - a recent paper . |
| Approach: | They propose a temporal reasoning dataset that evaluates the degree to which systems understand implicit events. |
| Outcome: | The proposed model outperforms baseline systems on TRACIE by 5% and 11% on MATRES, an explicit event benchmark. |
UNQOVERing Stereotyping Biases via Underspecified Questions (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Language embeddings have been shown to have stereotyping biases, but how these biase affecting downstream question answering models remains unexplored. |
| Approach: | They propose a general framework to probe biases through underspecified questions by building minimal context and building minimal questions. |
| Outcome: | The proposed framework isolates two types of reasoning errors and identifies stereotyping biases in gender, nationality, ethnicity, and religion classes. |
Is Multihop QA in DiRe Condition? Measuring and Reducing Disconnected Reasoning (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing models exploit dataset artifacts to produce correct answers without connecting information across multiple facts. |
| Approach: | They formalize disconnected reasoning across subsets of supporting facts to reduce disconnected reasoning . they propose an automatic transformation of existing datasets that reduces disconnected reasoning. |
| Outcome: | The proposed model-agnostic probe reduces disconnected reasoning in a reading comprehension setting. |
Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Prompts (2022.naacl-main)
Copied to clipboard
Daniel Khashabi, Xinxi Lyu, Sewon Min, Lianhui Qin, Kyle Richardson, Sean Welleck, Hannaneh Hajishirzi, Tushar Khot, Ashish Sabharwal, Sameer Singh, Yejin Choi
| Challenge: | Recent work shows the surprising power of continuous prompts to language models for controlled generation and solving a wide range of tasks. |
| Approach: | They propose to extract a discrete (textual) interpretation of continuous prompts faithful to the problem they solve. |
| Outcome: | The proposed model can find prompts that solve a task while being projected to an arbitrary text with a smaller drop in accuracy. |
What’s Missing: A Knowledge Gap Guided Approach for Multi-hop Question Answering (D19-1)
Copied to clipboard
| Challenge: | Multi-hop textual question answering requires combining information from multiple sentences. |
| Approach: | They propose a model that explicitly identifies the knowledge gap between a key span in the provided knowledge and the answer choices. |
| Outcome: | The proposed model outperforms existing models on the OpenBookQA dataset. |
A Simple Yet Strong Pipeline for HotpotQA (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for multi-hop question answering have been proposed with varying complexities. |
| Approach: | They propose to use BERT to identify potentially relevant sentences independently of each other . they feed selected sentences into a standard BERT span prediction model to choose an answer . |
| Outcome: | The proposed pipeline outperforms existing models on hotpotQA and support identification. |
Repurposing Entailment for Multi-Hop Question Answering Tasks (N19-1)
Copied to clipboard
| Challenge: | Existing approaches to use entailment models for question answering are limited . large scale datasets are typically framed at a sentence level, whereas question answering requires verifying whether multiple sentences, taken together as a premise, entitle a hypothesis. |
| Approach: | They propose a general architecture that can use entailment models for multi-hop QA tasks. |
| Outcome: | The proposed model outperforms QA models trained on target datasets and the OpenAI transformer models. |
Bridging Knowledge Gaps in Neural Entailment via Symbolic Models (D18-1)
Copied to clipboard
| Challenge: | Textual entailment models focus on lexical gaps but rarely on knowledge gaps. |
| Approach: | They propose a fact-level decomposition of the hypothesis and a knowledge lookup module to fill knowledge gaps in Science Entailment task. |
| Outcome: | The proposed model outperforms the base model on the SciTail dataset by 3% and 5% on the textual premise and the structured knowledge base. |
Exploiting Explicit Paths for Multi-hop Reading Comprehension (P19-1)
Copied to clipboard
| Challenge: | Existing approaches to multi-hop reading comprehension do not include multiple sentences or passages. |
| Approach: | They propose a path-based reasoning approach for a multi-hop reading comprehension task . they propose to extract paths from text and compose them to encode them . |
| Outcome: | The proposed model outperforms previous models on the multi-hop Wikihop dataset and can be generalized to the OpenBookQA dataset. |
♫ MuSiQue: Multihop Questions via Single-hop Question Composition (2022.tacl-1)
Copied to clipboard
| Challenge: | Existing multihop reasoning benchmarks are largely solvable via shortcuts . a bottom–up approach allows us to create a multihop QA dataset that requires proper multihop thinking. |
| Approach: | They propose a bottom–up approach that selects composable pairs of single-hop questions that are connected and adds stringent filters to the construction process. |
| Outcome: | The proposed approach creates a multihop question answering dataset with 25K 2–4 hop questions. |
SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories (2024.emnlp-main)
Copied to clipboard
Ben Bogin, Kejuan Yang, Shashank Gupta, Kyle Richardson, Erin Bransom, Peter Clark, Ashish Sabharwal, Tushar Khot
| Challenge: | Large Language Models (LLMs) have made significant progress in writing code, but can they be used to reproduce results from research repositories? |
| Approach: | They propose a benchmark to evaluate the capability of Large Language Models to reproduce results from research repositories. |
| Outcome: | The benchmark aims to capture the realistic challenges faced by researchers working with machine learning and natural language processing repositories. |
Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions (2023.acl-long)
Copied to clipboard
| Challenge: | Large language models generate natural language reasoning steps or Chains-of-Thoughts when prompted appropriately. |
| Approach: | They propose a new approach that interleaves retrieval with steps (sentences) in a CoT and uses retrieved results to improve CoT. |
| Outcome: | The proposed approach improves retrieval and downstream QA significantly on four datasets. |
ReadOnce Transformers: Reusable Representations of Text for Transformers (2021.acl-long)
Copied to clipboard
| Challenge: | ReadOnce Transformers is a task-independent, task-dependent, and compressed representation of text. |
| Approach: | They propose a transformer-based model that can build an information-capturing, task-independent, and compressed representation of text. |
| Outcome: | The proposed model can build an information-capturing, task-independent, and compressed representation of text. |
The Tail Wagging the Dog: Dataset Construction Biases of Social Bias Benchmarks (2023.acl-short)
Copied to clipboard
| Challenge: | omnipresence of large pre-trained language models has fueled concerns regarding systematic biases carried over from underlying data into the applications they are used in. |
| Approach: | They propose to compare social biases with non-social biase masked by alternate constructions that maintain the essence of their social bias. |
| Outcome: | The proposed benchmarks underestimate or overestimate the social bias in a given model. |
Hey AI, Can You Solve Complex Tasks by Talking to Agents? (2022.findings-acl)
Copied to clipboard
| Challenge: | CommaQA is a benchmark for learning to solve complex tasks by communicating with existing agents in natural language. |
| Approach: | They propose a synthetic benchmark with three complex reasoning tasks designed to be solved by communicating with existing QA agents. |
| Outcome: | The proposed model outperforms models that learn to communicate with agents without auxiliary supervision or data. |
Latent Factor Models Meets Instructions: Goal-conditioned Latent Factor Discovery without Task Supervision (2025.naacl-long)
Copied to clipboard
Zhouhang Xie, Tushar Khot, Bhavana Dalvi Mishra, Harshit Surana, Julian McAuley, Peter Clark, Bodhisattwa Prasad Majumder
| Challenge: | Instruction-following LLMs have recently allowed systems to discover hidden concepts from a collection of unstructured documents based on a natural language description of the purpose of the discovery (i.e., goal). |
| Approach: | They propose a goal-oriented latent factor discovery system that integrates LLM’s instruction-following ability with statistical models to handle large, noisy datasets where LLM reasoning alone falls short. |
| Outcome: | The proposed system improves task performance by 5-52% over baselines and 1.8 times as often as the best alternative, on average, in human evaluation. |
More Bang for Your Buck: Natural Perturbation for Robust Question Answering (2020.emnlp-main)
Copied to clipboard
| Challenge: | Creating large datasets to train NLP models is becoming increasingly expensive. |
| Approach: | They propose to use a question-answering dataset to expand a training set using human-driven perturbations instead of rule-based machine perturbations. |
| Outcome: | The proposed approach improves on a question-answering dataset with human-driven perturbations. |
Ethical-Advice Taker: Do Language Models Understand Natural Language Interventions? (2021.findings-acl)
Copied to clipboard
| Challenge: | a new task is proposed to modify a model's unethical behavior by communicating context-specific principles of ethics and equity to it. |
| Approach: | They propose a task to amend a questionanswering model's unethical behavior . they use social stereotypes to augment social stereotype models with ethical interventions . |
| Outcome: | The proposed task improves model behavior but fails to generalize to new domains . it is based on a zero-shot evaluation of social stereotypes in reading comprehension systems . |
GooAQ: Open Question Answering with Diverse Answer Types (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing research on niche answer types, mainly short responses and, in a few cases, long responses, has failed to adequately address the answer diversity of questions. |
| Approach: | They propose to use Google's autocomplete feature to collect questions from a large-scale dataset with a variety of answer types to facilitate further research on improving QA with diverse response types. |
| Outcome: | The proposed model produces naturalistic questions that are short and expressed using simple language. |
Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering (D18-1)
Copied to clipboard
| Challenge: | Existing QA datasets focus on linguistic understanding, but OpenBookQA probes deeper understanding of topic and language. |
| Approach: | They propose a dataset modeled after open book exams for question answering . the open book is a set of 1326 elementary level science facts . human performance on OpenBookQA is close to 92%, they show . |
| Outcome: | The proposed dataset is modeled after open book exams for question answering . human performance on OpenBookQA is close to 92%, but many state-of-the-art QA methods perform poorly . |
AdvEntuRe: Adversarial Training for Textual Entailment with Knowledge-Guided Examples (P18-1)
Copied to clipboard
| Challenge: | Recent deep learning entailment systems have achieved close to human level performance on large datasets, but the problem is far from solved. |
| Approach: | They propose a knowledge-guided adversarial example generator for incorporating large lexical resources into entailment models via only a handful of rule templates and a natural language example generator that iteratively adjusts to the discriminator’s weaknesses. |
| Outcome: | The proposed methods increase accuracy by 4.7% on SciTail and 2.8% on a 1% sub-sample of SNLI. |
Teaching Broad Reasoning Skills for Multi-Step QA by Generating Hard Contexts (2022.emnlp-main)
Copied to clipboard
| Challenge: | Question-answering datasets require a broad set of reasoning skills. |
| Approach: | They use QDMR representations to programmatically create hard-to-cheat synthetic contexts for real questions in multi-step reasoning datasets. |
| Outcome: | The proposed model improves performance by 13 F1 points on 4 multi-step QA datasets and 21 points on more complex questions. |
Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing approaches to decompose complex tasks into simpler ones do not require annotated decompositions. |
| Approach: | They propose a framework for building interpretable systems that learn to solve complex tasks by decomposing existing models into simpler ones solvable by existing models. |
| Outcome: | The proposed framework is more versatile than existing explainable systems for DROP and HotpotQA datasets, is more robust than state-of-the-art blackbox (uninterpretable) systems, and generates more understandable and trustworthy explanations compared to prior work. |