Papers by Oyvind Tafjord
UNIFIEDQA: Crossing Format Boundaries with a Single QA System (2020.findings-emnlp)
Copied to clipboard
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, Hannaneh Hajishirzi
| Challenge: | Question answering (QA) tasks have been posed using a variety of formats . a new study aims to develop specialized QA models that can be used to train QA systems . |
| Approach: | They build a pre-trained question answering model that performs well across 19 QA datasets . they argue that format-specialized models can limit the ability to teach reasoning . |
| Outcome: | a new model that trains on QA datasets performs on par with 8 models trained on individual datasets . a single model that trained on UNIFIEDQA performs well on 19 QA data . |
Digital Socrates: Evaluating LLMs through Explanation Critiques (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) can provide reasoned explanations, but the nature and quality of those explanations are still poorly understood. |
| Approach: | They propose to define a task of explanation critiquing and train an open-source automatic critique model using this data. |
| Outcome: | The proposed model can provide high-quality, nuanced evaluations without expensive API calls or human annotations. |
QuaRTz: An Open-Domain Dataset of Qualitative Relationship Questions (D19-1)
Copied to clipboard
| Challenge: | Unlike previous datasets, the general knowledge is textual and not tied to a fixed set of relationships. |
| Approach: | They introduce the first open-domain dataset, called QuaRTz, for reasoning about textual qualitative relationships. |
| Outcome: | The proposed dataset is the first open-domain dataset for reasoning about qualitative relationships. |
Multi-class Hierarchical Question Classification for Multiple Choice Science Exams (2020.lrec-1)
Copied to clipboard
Dongfang Xu, Peter Jansen, Jaycie Martin, Zhengnan Xie, Vikas Yadav, Harish Tayyar Madabushi, Oyvind Tafjord, Peter Clark
| Challenge: | Prior work has demonstrated that question classification (QC) can help answer a question more accurately. |
| Approach: | They propose to use a large dataset for question classification (QC) that contains 7,787 science exam questions paired with detailed classification labels from a fine-grained hierarchical taxonomy of 406 problem domains to train a BERT-based model. |
| Outcome: | The proposed model achieves a large (+0.12 MAP) gain while also achieving state-of-the-art performance on benchmark open-domain and biomedical QC datasets. |
OLMES: A Standard for Language Model Evaluations (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing models claim to perform better on tasks measuring model capabilities, but there is no standard setup for reproducible evaluations. |
| Approach: | They propose a document that is documented and practical for reproducible LLM evaluations and includes recommendations from existing literature and new experiments. |
| Outcome: | The proposed standard identifies and reviews the varying factors in evaluation practices adopted by the community, such as prompt formatting, choice of in-context examples, probability normalizations, and task formulation. |
Language Models with Rationality (2023.emnlp-main)
Copied to clipboard
| Challenge: | lack of interpretability is a growing impediment to widespread use of large language models . a new approach to solve this problem is to add a rational layer on top of the LLM . |
| Approach: | They propose to add a rational layer to the large language models to make model beliefs explicit . they also propose to identify and minimize contradictions in the model belief graph . |
| Outcome: | a new approach improves consistency without harming overall answer accuracy . the proposed approach makes model beliefs explicit and resolves inconsistencies . |
Reasoning Over Paragraph Effects in Situations (D19-58)
Copied to clipboard
| Challenge: | a key component of reading a passage is the ability to apply knowledge gained from the passage to a new situation. |
| Approach: | They propose a benchmark for reading comprehension targeting Reasoning Over Paragraph Effects in Situations. |
| Outcome: | The proposed model performs slightly better than randomly guessing an answer of the correct type, but is below the human performance of 89.0%. |
Explaining Answers with Entailment Trees (2021.emnlp-main)
Copied to clipboard
Bhavana Dalvi, Peter Jansen, Oyvind Tafjord, Zhengnan Xie, Hannah Smith, Leighanna Pipatanangkura, Peter Clark
| Challenge: | ENTAILMENTBANK is the first dataset to contain multistep entailment trees. |
| Approach: | They propose to generate explanations in the form of entailment trees, a tree of multipremise entanglements steps from facts that are known to the hypothesis of interest. |
| Outcome: | The proposed model can generate explanations in the form of entailment trees . this is a tree of multipremise enttailment steps from facts known to the hypothesis of interest. |
Enhancing Systematic Decompositional Natural Language Inference Using Informal Logic (2024.emnlp-main)
Copied to clipboard
Nathaniel Weir, Kate Sanders, Orion Weller, Shreya Sharma, Dongwei Jiang, Zhengping Jiang, Bhavana Dalvi Mishra, Oyvind Tafjord, Peter Jansen, Peter Clark, Benjamin Van Durme
| Challenge: | Recent language models allow structured reasoning with text, but lack of a clear protocol for discerning entailment causes noisy datasets and limited performance gains. |
| Approach: | They propose a consistent approach to annotating decompositional entailment and evaluate its impact on LLM-based textual inference. |
| Outcome: | The proposed approach has higher internal consistency than prior decompositional entailment datasets and significantly improves proof quality and accuracy. |
LILA: A Unified Benchmark for Mathematical Reasoning (2022.emnlp-main)
Copied to clipboard
Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, Ashwin Kalyan
| Challenge: | Towards evaluating and improving AI systems in this domain, we propose a mathematical reasoning benchmark based on 23 diversetasks . |
| Approach: | They propose a mathematical reasoning benchmark that includes 23 diverse tasks . they extend the benchmark by collecting task instructions and solutions in the form of Python programs . |
| Outcome: | The proposed model improves on multi-tasking while the best performing model only achieves 60.40%. |
Towards Teachable Reasoning Systems: Using a Dynamic Memory of User Feedback for Continual System Improvement (2022.emnlp-main)
Copied to clipboard
| Challenge: | Using simulated feedback, our system (called TeachMe) continually improves with time, and without model retraining. |
| Approach: | They propose to augment a QA model with a dynamic memory of user feedback, containing user-supplied corrections toerroneous model beliefs that users identify during interaction. |
| Outcome: | The proposed system improves with time and without model retraining, and with real users, by 15% on a hidden test set after teaching. |
Increasing Probability Mass on Answer Choices Does Not Always Improve Accuracy (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained language models (LMs) are used to discriminate on multiple-choice tasks that place probability mass on vocabulary tokens that aren’t among the given answer choices. |
| Approach: | They propose a mathematical formalism for SFC which allows us to quantify and bound its impact for the first time. |
| Outcome: | The proposed method eliminates the impact of SFC in the majority of instances. |
“You are grounded!”: Latent Name Artifacts in Pre-trained Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models perpetuate biases originating in their training corpus to downstream models. |
| Approach: | They focus on the representations of given names in pre-trained language models and show that name perturbation can have an effect on downstream tasks. |
| Outcome: | The proposed model can be used to model the representation of given names in pre-trained language models on reading comprehension probes where name perturbation changes the model answers. |
“Let Your Characters Tell Their Story”: A Dataset for Character-Centric Narrative Understanding (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on character-centric understanding of narratives focus on understanding the characters in the narrative, but these studies are limited to understanding only certain aspects of characters. |
| Approach: | They propose a dataset of literary pieces and their summaries paired with descriptions of characters that appear in them that are used to facilitate character-centric narrative understanding. |
| Outcome: | The proposed dataset includes literary pieces and their summaries paired with descriptions of characters that appear in them. |
CodeScientist: End-to-End Semi-Automated Scientific Discovery with Code-based Experimentation (2025.findings-acl)
Copied to clipboard
Peter Jansen, Oyvind Tafjord, Marissa Radensky, Pao Siangliulue, Tom Hope, Bhavana Dalvi Mishra, Bodhisattwa Prasad Majumder, Daniel S Weld, Peter Clark
| Challenge: | Automated scientific discovery (ASD) systems are limited in their evaluation of software artifacts and large volumes of research artifs are typically evaluated using conference-style paper review with limited evaluation of code. |
| Approach: | They propose a novel ASD system that frames ideation and experiment construction as a form of genetic search jointly over combinations of research articles and codeblocks defining common actions in a domain. |
| Outcome: | The proposed system returns 19 discoveries on machine-generated ideas in the domain of agents and virtual environments. |
ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Language (2021.findings-acl)
Copied to clipboard
| Challenge: | Recent work shows that transformers can generate both implications of a theory and the natural language proofs that support them. |
| Approach: | They propose a generative model that generates both implications of a theory and natural language proofs that support them. |
| Outcome: | The proposed model generates both implications of a theory and the natural language proofs that support them. |
SUPP.AI: finding evidence for supplement-drug interactions (2020.acl-demos)
Copied to clipboard
Lucy Lu Wang, Oyvind Tafjord, Arman Cohan, Sarthak Jain, Sam Skjonsberg, Carissa Schoenick, Nick Botner, Waleed Ammar
| Challenge: | Dietary supplements are used by a large portion of the population, but information on their pharmacologic interactions is incomplete. |
| Approach: | They propose an application to search evidence sentences extracted from the literature to identify supplement-drug interactions. |
| Outcome: | The proposed model extracts supplement information and identifies interactions using labeled DDI data. |
BeliefBank: Adding Memory to a Pre-Trained Language Model for a Systematic Notion of Belief (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained language models can produce inconsistent answers when probed, even after specialized training. |
| Approach: | They propose to embed a pretrained language model in a broader system that includes an evolving, symbolic memory of beliefs that records but may modify the raw PTLM answers. |
| Outcome: | The proposed architecture improves belief consistency in the overall system by revising beliefs that clash with others and generating queries using known beliefs as context. |
Entailer: Answering Questions with Faithful and Truthful Chains of Reasoning (2022.emnlp-main)
Copied to clipboard
| Challenge: | a system that can show how its answers are implied by its own internal beliefs via a systematic chain of reasoning would allow better understanding of why a model produced the answer it did. |
| Approach: | They propose to combine a backward-chaining model with a verifier that checks that the model itself believes those premises through self-querying to generate multistep chains that are both faithful (the answer follows from the reasoning) |
| Outcome: | The proposed model generates chains that are faithful and truthful while maintaining answer accuracy. |