Papers by Ashish Sabharwal

MoNaCo: More Natural and Complex Questions for Reasoning Across Dozens of Documents (2026.tacl-1)

Challenge:	Existing work on transformers established their Turing completeness, albeit with assumptions like infinite precision and arbitrarily powerful feedforward subnets.
Approach:	They propose to simulate transformers whose arithmetic precision is logarithmic in the number of input tokens by constant-depth logspace-uniform threshold circuits.
Outcome:	The proposed model emerges from the transformer architecture’s high parallelizability, suggesting a potential inherent weakness of the scaling paradigm.

Copied to clipboard

Tomer Wolfson, Harsh Trivedi, Mor Geva, Yoav Goldberg, Dan Roth, Tushar Khot, Ashish Sabharwal, Reut Tsarfaty

UNIFIEDQA: Crossing Format Boundaries with a Single QA System (2020.findings-emnlp)

Challenge:	Automated agents powered by large language models are becoming more ingrained into how people seek information . but evaluation benchmarks for LLMs rarely feature natural questions that are time-consuming . a new benchmark, MoNaCo, aims to address this gap by eliciting and manually answering time-wasting questions .
Approach:	They propose a benchmark of 1,315 natural and time-consuming questions that require dozens of intermediate steps to solve.
Outcome:	MoNaCo benchmarks achieve at least 61.2% F1 in real-world time-consuming questions hampered by low recall and hallucinations . Frontier LLMs evaluated on MoN achieving at least 61% F1, harmed by low memory and halluzinations.

Copied to clipboard

Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, Hannaneh Hajishirzi

DISCO: Distilling Counterfactuals with Large Language Models (2023.acl-long)

Challenge:	Question answering (QA) tasks have been posed using a variety of formats . a new study aims to develop specialized QA models that can be used to train QA systems .
Approach:	They build a pre-trained question answering model that performs well across 19 QA datasets . they argue that format-specialized models can limit the ability to teach reasoning .
Outcome:	a new model that trains on QA datasets performs on par with 8 models trained on individual datasets . a single model that trained on UNIFIEDQA performs well on 19 QA data .

Copied to clipboard

Zeming Chen, Qiyue Gao, Antoine Bosselut, Ashish Sabharwal, Kyle Richardson

ADaPT: As-Needed Decomposition and Planning with Language Models (2024.findings-naacl)

Challenge:	high-quality counterfactual data is scarce for most tasks and not easily generated at scale.
Approach:	They propose a method for automatically generating high-quality counterfactual data at scale . they use a large general language model to generate phrasal perturbations and filter them .
Outcome:	The proposed method is task-agnostic and can be applied to the task of natural language inference.

Copied to clipboard

Archiki Prasad, Alexander Koller, Mareike Hartmann, Peter Clark, Ashish Sabharwal, Mohit Bansal, Tushar Khot

Temporal Reasoning on Implicit Events from Distant Supervision (2021.naacl-main)

Challenge:	Large Language Models (LLMs) are increasingly being used for interactive decision-making tasks requiring planning and adapting to the environment.
Approach:	They propose an approach that explicitly plans and decomposes complex sub-tasks when the LLM is unable to execute them.
Outcome:	The proposed approach significantly outperforms established strong baselines, achieving success rates up to 28.3% higher in ALFWorld, 27% in WebShop, and 33% in TextCraft.

Copied to clipboard

Ben Zhou, Kyle Richardson, Qiang Ning, Tushar Khot, Ashish Sabharwal, Dan Roth

UNQOVERing Stereotyping Biases via Underspecified Questions (2020.findings-emnlp)

Challenge:	a novel temporal reasoning dataset evaluates the degree to which systems understand implicit events . state-of-the-art models struggle when predicting temporal relationships between implicit and explicit events - a recent paper .
Approach:	They propose a temporal reasoning dataset that evaluates the degree to which systems understand implicit events.
Outcome:	The proposed model outperforms baseline systems on TRACIE by 5% and 11% on MATRES, an explicit event benchmark.

Copied to clipboard

Tao Li, Daniel Khashabi, Tushar Khot, Ashish Sabharwal, Vivek Srikumar

Language Models with Rationality (2023.emnlp-main)

Challenge:	Language embeddings have been shown to have stereotyping biases, but how these biase affecting downstream question answering models remains unexplored.
Approach:	They propose a general framework to probe biases through underspecified questions by building minimal context and building minimal questions.
Outcome:	The proposed framework isolates two types of reasoning errors and identifies stereotyping biases in gender, nationality, ethnicity, and religion classes.

Copied to clipboard

Nora Kassner, Oyvind Tafjord, Ashish Sabharwal, Kyle Richardson, Hinrich Schuetze, Peter Clark

Is Multihop QA in DiRe Condition? Measuring and Reducing Disconnected Reasoning (2020.emnlp-main)

Challenge:	lack of interpretability is a growing impediment to widespread use of large language models . a new approach to solve this problem is to add a rational layer on top of the LLM .
Approach:	They propose to add a rational layer to the large language models to make model beliefs explicit . they also propose to identify and minimize contradictions in the model belief graph .
Outcome:	a new approach improves consistency without harming overall answer accuracy . the proposed approach makes model beliefs explicit and resolves inconsistencies .

Copied to clipboard

Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, Ashish Sabharwal

Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Prompts (2022.naacl-main)

Challenge:	Existing models exploit dataset artifacts to produce correct answers without connecting information across multiple facts.
Approach:	They formalize disconnected reasoning across subsets of supporting facts to reduce disconnected reasoning . they propose an automatic transformation of existing datasets that reduces disconnected reasoning.
Outcome:	The proposed model-agnostic probe reduces disconnected reasoning in a reading comprehension setting.

Copied to clipboard

Daniel Khashabi, Xinxi Lyu, Sewon Min, Lianhui Qin, Kyle Richardson, Sean Welleck, Hannaneh Hajishirzi, Tushar Khot, Ashish Sabharwal, Sameer Singh, Yejin Choi

How much coffee was consumed during EMNLP 2019? Fermi Problems: A New Reasoning Challenge for AI (2021.emnlp-main)

Challenge:	Recent work shows the surprising power of continuous prompts to language models for controlled generation and solving a wide range of tasks.
Approach:	They propose to extract a discrete (textual) interpretation of continuous prompts faithful to the problem they solve.
Outcome:	The proposed model can find prompts that solve a task while being projected to an arbitrary text with a smaller drop in accuracy.

Copied to clipboard

Ashwin Kalyan, Abhinav Kumar, Arjun Chandrasekaran, Ashish Sabharwal, Peter Clark

Leveraging Code to Improve In-Context Learning for Semantic Parsing (2024.naacl-long)

Challenge:	a new reasoning challenge is proposed to help AI systems to solve real-world problems . Fermi Problems are questions whose answers can only be approximated because their computation is either impossible or impossible.
Approach:	They propose a new reasoning challenge, Fermi Problems, which asks questions whose answers can only be approximated because their computation is either impractical or impossible.
Outcome:	The proposed datasets show that even fine-tuned large-scale language models perform poorly on these datasets.

Copied to clipboard

Ben Bogin, Shivanshu Gupta, Peter Clark, Ashish Sabharwal

Semantic Text Processing

Challenge:	In-context learning is an attractive approach for semantic parsing, but learning to parse to rare domain-specific languages from a few demonstrations is challenging.
Approach:	They propose to use Python instead of DSLs to augment prompts with a structured domain description.
Outcome:	The proposed approach improves accuracy and generalization across three datasets.

What’s Missing: A Knowledge Gap Guided Approach for Multi-hop Question Answering (D19-1)

Copied to clipboard

Tushar Khot, Ashish Sabharwal, Peter Clark

Breakpoint Transformers for Modeling and Tracking Intermediate Beliefs (2022.emnlp-main)

Challenge:	Multi-hop textual question answering requires combining information from multiple sentences.
Approach:	They propose a model that explicitly identifies the knowledge gap between a key span in the provided knowledge and the answer choices.
Outcome:	The proposed model outperforms existing models on the OpenBookQA dataset.

Copied to clipboard

Kyle Richardson, Ronen Tamari, Oren Sultan, Dafna Shahaf, Reut Tsarfaty, Ashish Sabharwal

Semantic Text Processing

Challenge:	a framework for language understanding models to track and improve beliefs through intermediate points in text is needed . breakpoint modeling is an efficient and end-to-end learning approach that trains models to train beliefs . understanding the behavior of models remains a formidable challenge for model safety, authors say .
Approach:	They propose a framework that trains models to track beliefs through intermediate points in text . their framework allows for efficient and robust learning of this type of model .
Outcome:	The proposed model outperforms strong representation learning approaches on a variety of NLU tasks.

A Simple Yet Strong Pipeline for HotpotQA (2020.emnlp-main)

Copied to clipboard

Dirk Groeneveld, Tushar Khot, null Mausam, Ashish Sabharwal

Repurposing Entailment for Multi-Hop Question Answering Tasks (N19-1)

Challenge:	Existing models for multi-hop question answering have been proposed with varying complexities.
Approach:	They propose to use BERT to identify potentially relevant sentences independently of each other . they feed selected sentences into a standard BERT span prediction model to choose an answer .
Outcome:	The proposed pipeline outperforms existing models on hotpotQA and support identification.

Copied to clipboard

Harsh Trivedi, Heeyoung Kwon, Tushar Khot, Ashish Sabharwal, Niranjan Balasubramanian

Saturated Transformers are Constant-Depth Threshold Circuits (2022.tacl-1)

Challenge:	Existing approaches to use entailment models for question answering are limited . large scale datasets are typically framed at a sentence level, whereas question answering requires verifying whether multiple sentences, taken together as a premise, entitle a hypothesis.
Approach:	They propose a general architecture that can use entailment models for multi-hop QA tasks.
Outcome:	The proposed model outperforms QA models trained on target datasets and the OpenAI transformer models.

Copied to clipboard

William Merrill, Ashish Sabharwal, Noah A. Smith

Bridging Knowledge Gaps in Neural Entailment via Symbolic Models (D18-1)

Challenge:	Recent work shows that transformers with hard attention are limited in power, but hard attention is a strong assumption.
Approach:	They propose a generalization of hard attention that captures attention patterns in transformers with saturated attention.
Outcome:	The proposed generalization of hard attention more closely captures the attention patterns learnable in practical transformers.

Copied to clipboard

Dongyeop Kang, Tushar Khot, Ashish Sabharwal, Peter Clark

Exploiting Explicit Paths for Multi-hop Reading Comprehension (P19-1)

Challenge:	Textual entailment models focus on lexical gaps but rarely on knowledge gaps.
Approach:	They propose a fact-level decomposition of the hypothesis and a knowledge lookup module to fill knowledge gaps in Science Entailment task.
Outcome:	The proposed model outperforms the base model on the SciTail dataset by 3% and 5% on the textual premise and the structured knowledge base.

Copied to clipboard

Souvik Kundu, Tushar Khot, Ashish Sabharwal, Peter Clark

♫ MuSiQue: Multihop Questions via Single-hop Question Composition (2022.tacl-1)

Challenge:	Existing approaches to multi-hop reading comprehension do not include multiple sentences or passages.
Approach:	They propose a path-based reasoning approach for a multi-hop reading comprehension task . they propose to extract paths from text and compose them to encode them .
Outcome:	The proposed model outperforms previous models on the multi-hop Wikihop dataset and can be generalized to the OpenBookQA dataset.

Copied to clipboard

Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, Ashish Sabharwal

LILA: A Unified Benchmark for Mathematical Reasoning (2022.emnlp-main)

Challenge:	Existing multihop reasoning benchmarks are largely solvable via shortcuts . a bottom–up approach allows us to create a multihop QA dataset that requires proper multihop thinking.
Approach:	They propose a bottom–up approach that selects composable pairs of single-hop questions that are connected and adds stringent filters to the construction process.
Outcome:	The proposed approach creates a multihop question answering dataset with 25K 2–4 hop questions.

Copied to clipboard

Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, Ashwin Kalyan

SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories (2024.emnlp-main)

Challenge:	Towards evaluating and improving AI systems in this domain, we propose a mathematical reasoning benchmark based on 23 diversetasks .
Approach:	They propose a mathematical reasoning benchmark that includes 23 diverse tasks . they extend the benchmark by collecting task instructions and solutions in the form of Python programs .
Outcome:	The proposed model improves on multi-tasking while the best performing model only achieves 60.40%.

Copied to clipboard

Ben Bogin, Kejuan Yang, Shashank Gupta, Kyle Richardson, Erin Bransom, Peter Clark, Ashish Sabharwal, Tushar Khot

Challenge:	Large Language Models (LLMs) have made significant progress in writing code, but can they be used to reproduce results from research repositories?
Approach:	They propose a benchmark to evaluate the capability of Large Language Models to reproduce results from research repositories.
Outcome:	The benchmark aims to capture the realistic challenges faced by researchers working with machine learning and natural language processing repositories.

Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions (2023.acl-long)

Copied to clipboard

Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, Ashish Sabharwal

Information Retrieval

Challenge:	Large language models generate natural language reasoning steps or Chains-of-Thoughts when prompted appropriately.
Approach:	They propose a new approach that interleaves retrieval with steps (sentences) in a CoT and uses retrieved results to improve CoT.
Outcome:	The proposed approach improves retrieval and downstream QA significantly on four datasets.

ReadOnce Transformers: Reusable Representations of Text for Transformers (2021.acl-long)

Copied to clipboard

Shih-Ting Lin, Ashish Sabharwal, Tushar Khot

Semantic Text Processing

Challenge:	ReadOnce Transformers is a task-independent, task-dependent, and compressed representation of text.
Approach:	They propose a transformer-based model that can build an information-capturing, task-independent, and compressed representation of text.
Outcome:	The proposed model can build an information-capturing, task-independent, and compressed representation of text.

Increasing Probability Mass on Answer Choices Does Not Always Improve Accuracy (2023.emnlp-main)

Copied to clipboard

Sarah Wiegreffe, Matthew Finlayson, Oyvind Tafjord, Peter Clark, Ashish Sabharwal

What Does My QA Model Know? Devising Controlled Probes Using Expert Knowledge (2020.tacl-1)

Challenge:	Pretrained language models (LMs) are used to discriminate on multiple-choice tasks that place probability mass on vocabulary tokens that aren’t among the given answer choices.
Approach:	They propose a mathematical formalism for SFC which allows us to quantify and bound its impact for the first time.
Outcome:	The proposed method eliminates the impact of SFC in the majority of instances.

Copied to clipboard

Kyle Richardson, Ashish Sabharwal

Not All Claims are Created Equal: Choosing the Right Statistical Approach to Assess Hypotheses (2020.acl-main)

Challenge:	Existing models are far from perfect when assessed at the level of clusters of semantically connected probes, such as all hypernym questions about a single concept.
Approach:	They propose a method for automatically building probe datasets from expert knowledge sources, allowing systematic control and a comprehensive evaluation.
Outcome:	The proposed model is predisposed to recognize certain types of structural linguistic knowledge, but performance degrades even with a slight increase in the number of “hops” in the underlying taxonomic hierarchy.

Copied to clipboard

Erfan Sadeqi Azer, Daniel Khashabi, Ashish Sabharwal, Dan Roth

Hey AI, Can You Solve Complex Tasks by Talking to Agents? (2022.findings-acl)

Challenge:	Empirical research in natural language processing has adopted a narrow set of principles for assessing hypotheses . alternative approaches to assess hypothese rely on p-value computation, which suffers from several known issues.
Approach:	They propose to compare different methods for assessing hypotheses . they argue that practitioners should first decide their target hypothesis before choosing a method .
Outcome:	The proposed method differs from other methods, but is not widely used in NLP . the proposed method is based on a p-value computation, but has a small gap in accuracy .

Copied to clipboard

Tushar Khot, Kyle Richardson, Daniel Khashabi, Ashish Sabharwal

Natural Language Interfaces

Challenge:	CommaQA is a benchmark for learning to solve complex tasks by communicating with existing agents in natural language.
Approach:	They propose a synthetic benchmark with three complex reasoning tasks designed to be solved by communicating with existing QA agents.
Outcome:	The proposed model outperforms models that learn to communicate with agents without auxiliary supervision or data.

IfQA: A Dataset for Open-domain Question Answering under Counterfactual Presuppositions (2023.emnlp-main)

Copied to clipboard

Wenhao Yu, Meng Jiang, Peter Clark, Ashish Sabharwal

More Bang for Your Buck: Natural Perturbation for Robust Question Answering (2020.emnlp-main)

Challenge:	Existing open-domain QA tasks focus on questions whose answer can be deduced directly from global factual knowledge.
Approach:	They propose a dataset where each question is based on a counterfactual presupposition via an "if" clause.
Outcome:	The IfQA dataset contains 3,800 questions that were annotated by crowdworkers on relevant Wikipedia passages.

Copied to clipboard

Daniel Khashabi, Tushar Khot, Ashish Sabharwal

Robustness in NLP

Challenge:	Creating large datasets to train NLP models is becoming increasingly expensive.
Approach:	They propose to use a question-answering dataset to expand a training set using human-driven perturbations instead of rule-based machine perturbations.
Outcome:	The proposed approach improves on a question-answering dataset with human-driven perturbations.

Ethical-Advice Taker: Do Language Models Understand Natural Language Interventions? (2021.findings-acl)

Copied to clipboard

Jieyu Zhao, Daniel Khashabi, Tushar Khot, Ashish Sabharwal, Kai-Wei Chang

GooAQ: Open Question Answering with Diverse Answer Types (2021.findings-emnlp)

Challenge:	a new task is proposed to modify a model's unethical behavior by communicating context-specific principles of ethics and equity to it.
Approach:	They propose a task to amend a questionanswering model's unethical behavior . they use social stereotypes to augment social stereotype models with ethical interventions .
Outcome:	The proposed task improves model behavior but fails to generalize to new domains . it is based on a zero-shot evaluation of social stereotypes in reading comprehension systems .

Copied to clipboard

Daniel Khashabi, Amos Ng, Tushar Khot, Ashish Sabharwal, Hannaneh Hajishirzi, Chris Callison-Burch

Transformers as Transducers (2025.tacl-1)

Challenge:	Existing research on niche answer types, mainly short responses and, in a few cases, long responses, has failed to adequately address the answer diversity of questions.
Approach:	They propose to use Google's autocomplete feature to collect questions from a large-scale dataset with a variety of answer types to facilitate further research on improving QA with diverse response types.
Outcome:	The proposed model produces naturalistic questions that are short and expressed using simple language.

Copied to clipboard

Lena Strobl, Dana Angluin, David Chiang, Jonathan Rawski, Ashish Sabharwal

Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering (D18-1)

Challenge:	Using finite transducers, we find that transformers can express large classes of (total functional) transductions.
Approach:	They extend existing RASP programming language to sequence-to-sequence transductions and introduce two new extensions.
Outcome:	The proposed model can express surprisingly large classes of (total functional) transductions.

Copied to clipboard

Todor Mihaylov, Peter Clark, Tushar Khot, Ashish Sabharwal

Information Retrieval

Challenge:	Existing QA datasets focus on linguistic understanding, but OpenBookQA probes deeper understanding of topic and language.
Approach:	They propose a dataset modeled after open book exams for question answering . the open book is a set of 1326 elementary level science facts . human performance on OpenBookQA is close to 92%, they show .
Outcome:	The proposed dataset is modeled after open book exams for question answering . human performance on OpenBookQA is close to 92%, but many state-of-the-art QA methods perform poorly .

AdvEntuRe: Adversarial Training for Textual Entailment with Knowledge-Guided Examples (P18-1)

Copied to clipboard

Dongyeop Kang, Tushar Khot, Ashish Sabharwal, Eduard Hovy

Teaching Broad Reasoning Skills for Multi-Step QA by Generating Hard Contexts (2022.emnlp-main)

Challenge:	Recent deep learning entailment systems have achieved close to human level performance on large datasets, but the problem is far from solved.
Approach:	They propose a knowledge-guided adversarial example generator for incorporating large lexical resources into entailment models via only a handful of rule templates and a natural language example generator that iteratively adjusts to the discriminator’s weaknesses.
Outcome:	The proposed methods increase accuracy by 4.7% on SciTail and 2.8% on a 1% sub-sample of SNLI.

Copied to clipboard

Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, Ashish Sabharwal

Robustness in NLP

Challenge:	Question-answering datasets require a broad set of reasoning skills.
Approach:	They use QDMR representations to programmatically create hard-to-cheat synthetic contexts for real questions in multi-step reasoning datasets.
Outcome:	The proposed model improves performance by 13 F1 points on 4 multi-step QA datasets and 21 points on more complex questions.

Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models (2021.naacl-main)

Copied to clipboard

Tushar Khot, Daniel Khashabi, Kyle Richardson, Peter Clark, Ashish Sabharwal