Papers by Peter Clark

Learning to repair: Repairing model output errors after deployment using a dynamic memory of feedback (2022.findings-naacl)

Challenge:	Question answering (QA) tasks have been posed using a variety of formats . a new study aims to develop specialized QA models that can be used to train QA systems .
Approach:	They build a pre-trained question answering model that performs well across 19 QA datasets . they argue that format-specialized models can limit the ability to teach reasoning .
Outcome:	a new model that trains on QA datasets performs on par with 8 models trained on individual datasets . a single model that trained on UNIFIEDQA performs well on 19 QA data .

Copied to clipboard

Niket Tandon, Aman Madaan, Peter Clark, Yiming Yang

ADaPT: As-Needed Decomposition and Planning with Language Models (2024.findings-naacl)

Challenge:	Our approach pairs an LM with a growing memory of cases where the user identified an output error and provided general feedback on how to correct it.
Approach:	They propose to use an existing script generator to train a model to repair output errors without retraining.
Outcome:	The proposed model learns to apply user feedback to repair output errors while avoiding similar past mistakes on new, unseen examples.

Copied to clipboard

Archiki Prasad, Alexander Koller, Mareike Hartmann, Peter Clark, Ashish Sabharwal, Mohit Bansal, Tushar Khot

Digital Socrates: Evaluating LLMs through Explanation Critiques (2024.acl-long)

Challenge:	Large Language Models (LLMs) are increasingly being used for interactive decision-making tasks requiring planning and adapting to the environment.
Approach:	They propose an approach that explicitly plans and decomposes complex sub-tasks when the LLM is unable to execute them.
Outcome:	The proposed approach significantly outperforms established strong baselines, achieving success rates up to 28.3% higher in ALFWorld, 27% in WebShop, and 33% in TextCraft.

Copied to clipboard

Yuling Gu, Oyvind Tafjord, Peter Clark

QuaRTz: An Open-Domain Dataset of Qualitative Relationship Questions (D19-1)

Challenge:	Large language models (LLMs) can provide reasoned explanations, but the nature and quality of those explanations are still poorly understood.
Approach:	They propose to define a task of explanation critiquing and train an open-source automatic critique model using this data.
Outcome:	The proposed model can provide high-quality, nuanced evaluations without expensive API calls or human annotations.

Copied to clipboard

Oyvind Tafjord, Matt Gardner, Kevin Lin, Peter Clark

Challenge:	Unlike previous datasets, the general knowledge is textual and not tied to a fixed set of relationships.
Approach:	They introduce the first open-domain dataset, called QuaRTz, for reasoning about textual qualitative relationships.
Outcome:	The proposed dataset is the first open-domain dataset for reasoning about qualitative relationships.

Multi-class Hierarchical Question Classification for Multiple Choice Science Exams (2020.lrec-1)

Copied to clipboard

Dongfang Xu, Peter Jansen, Jaycie Martin, Zhengnan Xie, Vikas Yadav, Harish Tayyar Madabushi, Oyvind Tafjord, Peter Clark

Information Retrieval

Challenge:	Prior work has demonstrated that question classification (QC) can help answer a question more accurately.
Approach:	They propose to use a large dataset for question classification (QC) that contains 7,787 science exam questions paired with detailed classification labels from a fine-grained hierarchical taxonomy of 406 problem domains to train a BERT-based model.
Outcome:	The proposed model achieves a large (+0.12 MAP) gain while also achieving state-of-the-art performance on benchmark open-domain and biomedical QC datasets.

What-if I ask you to explain: Explaining the effects of perturbations in procedural text (2020.findings-emnlp)

Copied to clipboard

Dheeraj Rajagopal, Niket Tandon, Peter Clark, Bhavana Dalvi, Eduard Hovy

Explainability & Interpretability in NLP

Challenge:	QUARTET constructs explanations from paragraphs using procedural text . qartet achieves 18 points better on explanation accuracy compared to strong baselines on a recent process comprehension benchmark.
Approach:	They propose a system that constructs explanations from paragraphs by modeling the explanation task as a multitask learning problem.
Outcome:	The proposed system achieves 18 points better on explanation accuracy compared to strong baselines on a process comprehension benchmark.

Tailoring with Targeted Precision: Edit-Based Agents for Open-Domain Procedure Customization (2024.findings-acl)

Copied to clipboard

Yash Kumar Lal, Li Zhang, Faeze Brahman, Bodhisattwa Prasad Majumder, Peter Clark, Niket Tandon

Challenge:	Using a set of over 200 WikiHow procedures, we test several simple multi-LLM-agent architectures for customization.
Approach:	They propose to use a set of WikiHow procedures to test how-to procedures can be customized by multiple LLMs.
Outcome:	The proposed architecture outperforms an end-to-end LLM in the evaluation set of over 200 WikiHow procedures.

Commonsense Inference in Natural Language Processing (COIN) - Shared Task Report (D19-60)

Copied to clipboard

Simon Ostermann, Sheng Zhang, Michael Roth, Peter Clark

Language Models with Rationality (2023.emnlp-main)

Challenge:	The workshop on Commonsense Inference in NLP (COIN) evaluated text understanding systems' ability to draw inferences about facts that are not mentioned in the text, but that are assumed to be common ground.
Approach:	They propose to use commonsense knowledge to evaluate systems' ability to answer questions/queries about a text.
Outcome:	The proposed tasks evaluated systems in two contexts: Commonsense Inference and Commonsensible Inference.

Copied to clipboard

Nora Kassner, Oyvind Tafjord, Ashish Sabharwal, Kyle Richardson, Hinrich Schuetze, Peter Clark

How much coffee was consumed during EMNLP 2019? Fermi Problems: A New Reasoning Challenge for AI (2021.emnlp-main)

Challenge:	lack of interpretability is a growing impediment to widespread use of large language models . a new approach to solve this problem is to add a rational layer on top of the LLM .
Approach:	They propose to add a rational layer to the large language models to make model beliefs explicit . they also propose to identify and minimize contradictions in the model belief graph .
Outcome:	a new approach improves consistency without harming overall answer accuracy . the proposed approach makes model beliefs explicit and resolves inconsistencies .

Copied to clipboard

Ashwin Kalyan, Abhinav Kumar, Arjun Chandrasekaran, Ashish Sabharwal, Peter Clark

Reasoning Over Paragraph Effects in Situations (D19-58)

Challenge:	a new reasoning challenge is proposed to help AI systems to solve real-world problems . Fermi Problems are questions whose answers can only be approximated because their computation is either impossible or impossible.
Approach:	They propose a new reasoning challenge, Fermi Problems, which asks questions whose answers can only be approximated because their computation is either impractical or impossible.
Outcome:	The proposed datasets show that even fine-tuned large-scale language models perform poorly on these datasets.

Copied to clipboard

Kevin Lin, Oyvind Tafjord, Peter Clark, Matt Gardner

Generating Literature-Driven Scientific Theories at Scale (2026.acl-long)

Challenge:	a key component of reading a passage is the ability to apply knowledge gained from the passage to a new situation.
Approach:	They propose a benchmark for reading comprehension targeting Reasoning Over Paragraph Effects in Situations.
Outcome:	The proposed model performs slightly better than randomly guessing an answer of the correct type, but is below the human performance of 89.0%.

Copied to clipboard

Peter Jansen, Peter Clark, Doug Downey, Daniel S Weld

Linguistic Theories

Challenge:	Contemporary automated scientific discovery systems focus on generating experiments, but higher-level activities such as theory building remain underexplored.
Approach:	They propose to synthesize theories from scientific literature using literature-grounding versus parametric knowledge.
Outcome:	The proposed method matches existing evidence better than parametric LLM memory generation.

Leveraging Code to Improve In-Context Learning for Semantic Parsing (2024.naacl-long)

Copied to clipboard

Ben Bogin, Shivanshu Gupta, Peter Clark, Ashish Sabharwal

What’s Missing: A Knowledge Gap Guided Approach for Multi-hop Question Answering (D19-1)

Challenge:	In-context learning is an attractive approach for semantic parsing, but learning to parse to rare domain-specific languages from a few demonstrations is challenging.
Approach:	They propose to use Python instead of DSLs to augment prompts with a structured domain description.
Outcome:	The proposed approach improves accuracy and generalization across three datasets.

Copied to clipboard

Tushar Khot, Ashish Sabharwal, Peter Clark

Let GPT be a Math Tutor: Teaching Math Word Problem Solvers with Customized Exercise Generation (2023.emnlp-main)

Challenge:	Multi-hop textual question answering requires combining information from multiple sentences.
Approach:	They propose a model that explicitly identifies the knowledge gap between a key span in the provided knowledge and the answer choices.
Outcome:	The proposed model outperforms existing models on the OpenBookQA dataset.

Copied to clipboard

Zhenwen Liang, Wenhao Yu, Tanmay Rajpurohit, Peter Clark, Xiangliang Zhang, Ashwin Kalyan

Numerical Reasoning

Challenge:	Existing approaches for distilling large language models into smaller, more efficient student models are based on educational science principles such as knowledge tracing and personalized learning.
Approach:	They propose a method for distilling large language models into smaller, more efficient student models that are aligned with educational science principles such as knowledge tracing and personalized learning.
Outcome:	The proposed approach outperforms LLMs on three benchmarks while employing significantly fewer parameters.

Bridging Knowledge Gaps in Neural Entailment via Symbolic Models (D18-1)

Copied to clipboard

Dongyeop Kang, Tushar Khot, Ashish Sabharwal, Peter Clark

Explaining Answers with Entailment Trees (2021.emnlp-main)

Challenge:	Textual entailment models focus on lexical gaps but rarely on knowledge gaps.
Approach:	They propose a fact-level decomposition of the hypothesis and a knowledge lookup module to fill knowledge gaps in Science Entailment task.
Outcome:	The proposed model outperforms the base model on the SciTail dataset by 3% and 5% on the textual premise and the structured knowledge base.

Copied to clipboard

Bhavana Dalvi, Peter Jansen, Oyvind Tafjord, Zhengnan Xie, Hannah Smith, Leighanna Pipatanangkura, Peter Clark

Enhancing Systematic Decompositional Natural Language Inference Using Informal Logic (2024.emnlp-main)

Challenge:	ENTAILMENTBANK is the first dataset to contain multistep entailment trees.
Approach:	They propose to generate explanations in the form of entailment trees, a tree of multipremise entanglements steps from facts that are known to the hypothesis of interest.
Outcome:	The proposed model can generate explanations in the form of entailment trees . this is a tree of multipremise enttailment steps from facts known to the hypothesis of interest.

Copied to clipboard

Nathaniel Weir, Kate Sanders, Orion Weller, Shreya Sharma, Dongwei Jiang, Zhengping Jiang, Bhavana Dalvi Mishra, Oyvind Tafjord, Peter Jansen, Peter Clark, Benjamin Van Durme

Textual Inference

Challenge:	Recent language models allow structured reasoning with text, but lack of a clear protocol for discerning entailment causes noisy datasets and limited performance gains.
Approach:	They propose a consistent approach to annotating decompositional entailment and evaluate its impact on LLM-based textual inference.
Outcome:	The proposed approach has higher internal consistency than prior decompositional entailment datasets and significantly improves proof quality and accuracy.

Exploiting Explicit Paths for Multi-hop Reading Comprehension (P19-1)

Copied to clipboard

Souvik Kundu, Tushar Khot, Ashish Sabharwal, Peter Clark

Be Consistent! Improving Procedural Text Comprehension using Label Consistency (N19-1)

Challenge:	Existing approaches to multi-hop reading comprehension do not include multiple sentences or passages.
Approach:	They propose a path-based reasoning approach for a multi-hop reading comprehension task . they propose to extract paths from text and compose them to encode them .
Outcome:	The proposed model outperforms previous models on the multi-hop Wikihop dataset and can be generalized to the OpenBookQA dataset.

Copied to clipboard

Xinya Du, Bhavana Dalvi, Niket Tandon, Antoine Bosselut, Wen-tau Yih, Peter Clark, Claire Cardie

Challenge:	Existing systems for procedural text comprehension still struggle with this task . evaluative work shows that consistent predictions from multiple entities can improve performance .
Approach:	They propose a framework that leverages label consistency during training to improve prediction performance.
Outcome:	The proposed framework significantly improves prediction performance over previous state-of-the-art systems on a standard benchmark dataset for procedural text, ProPara.

Do language models have coherent mental models of everyday things? (2023.acl-long)

Copied to clipboard

Yuling Gu, Bhavana Dalvi Mishra, Peter Clark

Learning to Explain: Datasets and Models for Identifying Valid Reasoning Chains in Multihop Question-Answering (2020.emnlp-main)

Challenge:	Psychologists and cognitive scientists hypothesize that humans develop mental models of the world, namely internal, conceptual representations of the environment which we base our decisions and actions on.
Approach:	They propose to add a constraint satisfaction layer to the LM's raw predictions to apply commonsense constraints to reduce incoherence.
Outcome:	The proposed extension removes inconsistencies and improves accuracy by 16-20%.

Copied to clipboard

Harsh Jhamtani, Peter Clark

PDDLEGO: Iterative Planning in Textual Environments (2024.starsem-1)

Challenge:	despite rapid progress in multihop question-answering, models still have trouble explaining why an answer is correct.
Approach:	They propose three explanation datasets in which explanations from corpus facts are annotated . they first annotate multiple candidate explanations for each answer, then use crowd-sourcing perturbations to test generalization .
Outcome:	The proposed datasets improve explanation quality but still behind the upper bound . the proposed dataset can be used to improve explanations using a BERT-based classifier .

Copied to clipboard

Li Zhang, Peter Jansen, Tianyi Zhang, Peter Clark, Chris Callison-Burch, Niket Tandon

Challenge:	Existing methods to plan in textual environments rely on a fully-observed environment where all entity states are known, but are not interpretable.
Approach:	They propose to use LLMs to generate a formal representation of the environment that can be solved by a symbolic planner.
Outcome:	The proposed model outperforms existing methods in the Coin Collector simulation and Cooking World simulations.

LILA: A Unified Benchmark for Mathematical Reasoning (2022.emnlp-main)

Copied to clipboard

Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, Ashwin Kalyan

Memory-assisted prompt editing to improve GPT-3 after deployment (2022.emnlp-main)

Challenge:	Towards evaluating and improving AI systems in this domain, we propose a mathematical reasoning benchmark based on 23 diversetasks .
Approach:	They propose a mathematical reasoning benchmark that includes 23 diverse tasks . they extend the benchmark by collecting task instructions and solutions in the form of Python programs .
Outcome:	The proposed model improves on multi-tasking while the best performing model only achieves 60.40%.

Copied to clipboard

Aman Madaan, Niket Tandon, Peter Clark, Yiming Yang

SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories (2024.emnlp-main)

Challenge:	Large LMs such as GPT-3 can commit mistakes that are obvious to humans, such as interpreting “What word is similar to good?” to mean a homophone, while the user intended a synonym.
Approach:	They pair GPT-3 with a growing memory of cases where the model misunderstood the user’s intents, along with user feedback for clarification.
Outcome:	The proposed model can correct misunderstandings on four lexical tasks and two advanced ethical reasoning tasks without retraining.

Copied to clipboard

Ben Bogin, Kejuan Yang, Shashank Gupta, Kyle Richardson, Erin Bransom, Peter Clark, Ashish Sabharwal, Tushar Khot

Challenge:	Large Language Models (LLMs) have made significant progress in writing code, but can they be used to reproduce results from research repositories?
Approach:	They propose a benchmark to evaluate the capability of Large Language Models to reproduce results from research repositories.
Outcome:	The benchmark aims to capture the realistic challenges faced by researchers working with machine learning and natural language processing repositories.

Towards Teachable Reasoning Systems: Using a Dynamic Memory of User Feedback for Continual System Improvement (2022.emnlp-main)

Copied to clipboard

Bhavana Dalvi Mishra, Oyvind Tafjord, Peter Clark

Increasing Probability Mass on Answer Choices Does Not Always Improve Accuracy (2023.emnlp-main)

Challenge:	Using simulated feedback, our system (called TeachMe) continually improves with time, and without model retraining.
Approach:	They propose to augment a QA model with a dynamic memory of user feedback, containing user-supplied corrections toerroneous model beliefs that users identify during interaction.
Outcome:	The proposed system improves with time and without model retraining, and with real users, by 15% on a hidden test set after teaching.

Copied to clipboard

Sarah Wiegreffe, Matthew Finlayson, Oyvind Tafjord, Peter Clark, Ashish Sabharwal

Think about it! Improving defeasible reasoning by first modeling the question scenario. (2021.emnlp-main)

Challenge:	Pretrained language models (LMs) are used to discriminate on multiple-choice tasks that place probability mass on vocabulary tokens that aren’t among the given answer choices.
Approach:	They propose a mathematical formalism for SFC which allows us to quantify and bound its impact for the first time.
Outcome:	The proposed method eliminates the impact of SFC in the majority of instances.

Copied to clipboard

Aman Madaan, Niket Tandon, Dheeraj Rajagopal, Peter Clark, Yiming Yang, Eduard Hovy

Multimodality

Challenge:	Existing literature suggests that a person forms a mental model of the problem scenario before answering questions.
Approach:	They propose to have a model first create a graph of relevant influences and leverage that graph as an additional input when answering a defeasible query.
Outcome:	The proposed model achieves state-of-the-art on three different defeasible reasoning datasets.

HypER: Literature-grounded Hypothesis Generation and Distillation with Provenance (2025.emnlp-main)

Copied to clipboard

Rosni Vasu, Chandrayee Basu, Bhavana Dalvi Mishra, Cristina Sarasua, Peter Clark, Abraham Bernstein

The Unreasonable Effectiveness of Easy Training Data for Hard Tasks (2024.acl-long)

Challenge:	Existing approaches focus on retrieval augmentation and focus on the quality of the output . Existing methods focus on generating a highly specific declarative statement ignoring the underlying reasoning process behind ideation.
Approach:	They propose a large language model that generates evidence-based hypotheses using literature-guided reasoning and a multi-task setting.
Outcome:	The proposed model outperforms the base model and generates evidence-grounded hypotheses with high feasibility and impact as judged by human experts.

Copied to clipboard

Peter Hase, Mohit Bansal, Peter Clark, Sarah Wiegreffe

DREAM: Improving Situational QA by First Elaborating the Situation (2022.naacl-main)

Challenge:	Existing pretrained language models perform well on hard data, but hard data is noisier and costlier to collect.
Approach:	They propose to use in-context learning, linear classifier heads, and QLoRA to show that pretrained language models generalize relatively well from easy to hard data.
Outcome:	The proposed model generalizes well from easy to hard data even better than oracle models finetuned on hard data.

Copied to clipboard

Yuling Gu, Bhavana Dalvi, Peter Clark

IfQA: A Dataset for Open-domain Question Answering under Counterfactual Presuppositions (2023.emnlp-main)

Challenge:	Cognitive science has long promoted the formation of mental models as central to understanding and question-answering.
Approach:	They train a new model, DREAM, to answer questions that elaborate the scenes that situated questions are about and then provide those elaborations as additional context to a question-answering (QA) model.
Outcome:	The proposed model is able to create better scene elaborations than a representative state-of-the-art, zero-shot model.

Copied to clipboard

Wenhao Yu, Meng Jiang, Peter Clark, Ashish Sabharwal

Reasoning about Actions and State Changes by Injecting Commonsense Knowledge (D18-1)

Challenge:	Existing open-domain QA tasks focus on questions whose answer can be deduced directly from global factual knowledge.
Approach:	They propose a dataset where each question is based on a counterfactual presupposition via an "if" clause.
Outcome:	The IfQA dataset contains 3,800 questions that were annotated by crowdworkers on relevant Wikipedia passages.

Copied to clipboard

Niket Tandon, Bhavana Dalvi, Joel Grus, Wen-tau Yih, Antoine Bosselut, Peter Clark

Latent Factor Models Meets Instructions: Goal-conditioned Latent Factor Discovery without Task Supervision (2025.naacl-long)

Challenge:	Recent work has shown impressive progress in comprehending procedural text, but their predictions can be inconsistent or highly improbable.
Approach:	They propose to incorporate global constraints and bias reading with corpora-based preferences to improve the predicted effects of actions in a paragraph.
Outcome:	The proposed model significantly outperforms earlier models on a benchmark dataset for procedural text comprehension (+8% relative gain) it avoids nonsensical predictions that earlier models make, and it is more robust than previous models.

Copied to clipboard

Zhouhang Xie, Tushar Khot, Bhavana Dalvi Mishra, Harshit Surana, Julian McAuley, Peter Clark, Bodhisattwa Prasad Majumder

CodeScientist: End-to-End Semi-Automated Scientific Discovery with Code-based Experimentation (2025.findings-acl)

Challenge:	Instruction-following LLMs have recently allowed systems to discover hidden concepts from a collection of unstructured documents based on a natural language description of the purpose of the discovery (i.e., goal).
Approach:	They propose a goal-oriented latent factor discovery system that integrates LLM’s instruction-following ability with statistical models to handle large, noisy datasets where LLM reasoning alone falls short.
Outcome:	The proposed system improves task performance by 5-52% over baselines and 1.8 times as often as the best alternative, on average, in human evaluation.

Copied to clipboard

Peter Jansen, Oyvind Tafjord, Marissa Radensky, Pao Siangliulue, Tom Hope, Bhavana Dalvi Mishra, Bodhisattwa Prasad Majumder, Daniel S Weld, Peter Clark

Challenge:	Automated scientific discovery (ASD) systems are limited in their evaluation of software artifacts and large volumes of research artifs are typically evaluated using conference-style paper review with limited evaluation of code.
Approach:	They propose a novel ASD system that frames ideation and experiment construction as a form of genetic search jointly over combinations of research articles and codeblocks defining common actions in a domain.
Outcome:	The proposed system returns 19 discoveries on machine-generated ideas in the domain of agents and virtual environments.

WIQA: A dataset for “What if...” reasoning over procedural text (D19-1)

Copied to clipboard

Niket Tandon, Bhavana Dalvi, Keisuke Sakaguchi, Peter Clark, Antoine Bosselut

ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Language (2021.findings-acl)

Challenge:	a dataset of “What if...” questions is available for procedural text comprehension . we present the dataset as an open challenge to the community .
Approach:	They propose a dataset of “What if...” questions over procedural text . they use paragraphs annotated with multiple influence graphs to create the questions .
Outcome:	The proposed dataset achieves 73.8% accuracy, well below the human performance of 96.3%.

Copied to clipboard

Oyvind Tafjord, Bhavana Dalvi, Peter Clark

Linguistics & Cognitive NLP

Challenge:	Recent work shows that transformers can generate both implications of a theory and the natural language proofs that support them.
Approach:	They propose a generative model that generates both implications of a theory and natural language proofs that support them.
Outcome:	The proposed model generates both implications of a theory and the natural language proofs that support them.

NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks (2022.acl-long)

Copied to clipboard

Swaroop Mishra, Arindam Mitra, Neeraj Varshney, Bhavdeep Sachdeva, Peter Clark, Chitta Baral, Ashwin Kalyan

BeliefBank: Adding Memory to a Pre-Trained Language Model for a Systematic Notion of Belief (2021.emnlp-main)

Challenge:	Existing AI systems fail to perform basic mathematical reasoning when presented in a slightly different manner.
Approach:	They propose a multi-task benchmark that evaluates the performance of AI systems on eight different tasks that at their core require simple arithmetic understanding.
Outcome:	The proposed benchmark compares the performance of AI systems on eight different tasks.

Copied to clipboard

Nora Kassner, Oyvind Tafjord, Hinrich Schütze, Peter Clark

Everything Happens for a Reason: Discovering the Purpose of Actions in Procedural Text (D19-1)

Challenge:	Pretrained language models can produce inconsistent answers when probed, even after specialized training.
Approach:	They propose to embed a pretrained language model in a broader system that includes an evolving, symbolic memory of beliefs that records but may modify the raw PTLM answers.
Outcome:	The proposed architecture improves belief consistency in the overall system by revising beliefs that clash with others and generating queries using known beliefs as context.

Copied to clipboard

Bhavana Dalvi, Niket Tandon, Antoine Bosselut, Wen-tau Yih, Peter Clark

Can Language Models Serve as Text-Based World Simulators? (2024.acl-short)

Challenge:	XPAD is a new model that predicts actions' effects and their dependencies based on background knowledge . previous work on extracting sequences of actions from text has focused on identifying why they are the way they are .
Approach:	They propose a new model that biases effect predictions towards those that explain more of the actions in the paragraph and are more plausible with respect to background knowledge.
Outcome:	The proposed model outperforms existing systems on explaining actions by predicting dependencies while maintaining the performance on the original task in ProPara.

Copied to clipboard

Ruoyao Wang, Graham Todd, Ziang Xiao, Xingdi Yuan, Marc-Alexandre Côté, Peter Clark, Peter Jansen

RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputs (2023.acl-long)

Challenge:	Recent advances in large language models (LLMs) have pointed towards an alternative approach by leveraging the huge amount of knowledge contained in their pre-training datasets.
Approach:	They build and use a benchmark to quantify how well text-based simulators can serve as text-driven world simulators.
Outcome:	The proposed benchmark aims to quantify how well language models can serve as world simulators.

Copied to clipboard

Afra Feyza Akyurek, Ekin Akyurek, Ashwin Kalyan, Peter Clark, Derry Tanti Wijaya, Niket Tandon

Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering (D18-1)

Challenge:	Despite their success, even the largest language models make mistakes.
Approach:	They propose a framework where one language model can generate critiques to improve its peer's performance.
Outcome:	The proposed framework improves the performance of a fixed model 200 times its size by 10% over other models.

Copied to clipboard

Todor Mihaylov, Peter Clark, Tushar Khot, Ashish Sabharwal

Information Retrieval

Challenge:	Existing QA datasets focus on linguistic understanding, but OpenBookQA probes deeper understanding of topic and language.
Approach:	They propose a dataset modeled after open book exams for question answering . the open book is a set of 1326 elementary level science facts . human performance on OpenBookQA is close to 92%, they show .
Outcome:	The proposed dataset is modeled after open book exams for question answering . human performance on OpenBookQA is close to 92%, but many state-of-the-art QA methods perform poorly .

Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models (2021.naacl-main)

Copied to clipboard

Tushar Khot, Daniel Khashabi, Kyle Richardson, Peter Clark, Ashish Sabharwal

Entailer: Answering Questions with Faithful and Truthful Chains of Reasoning (2022.emnlp-main)

Challenge:	Existing approaches to decompose complex tasks into simpler ones do not require annotated decompositions.
Approach:	They propose a framework for building interpretable systems that learn to solve complex tasks by decomposing existing models into simpler ones solvable by existing models.
Outcome:	The proposed framework is more versatile than existing explainable systems for DROP and HotpotQA datasets, is more robust than state-of-the-art blackbox (uninterpretable) systems, and generates more understandable and trustworthy explanations compared to prior work.

Copied to clipboard

Oyvind Tafjord, Bhavana Dalvi Mishra, Peter Clark