Papers by Peter Jansen
Self-Supervised Behavior Cloned Transformers are Path Crawlers for Text Games (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Text game agents are often modeled using reinforcement learning, but their performance is limited. |
| Approach: | They propose a self-supervised behavior cloning transformer for text games . they explore trajectories that lead to reward within the games and then train small models . their approach consistently uncovers generalizable training data, achieving 90% performance of supervised systems across three benchmark text games. |
| Outcome: | The proposed model achieves 90% performance on three text games. |
CodeDistiller: Automatically Generating Code Libraries for Scientific Coding Agents (2026.acl-demo)
Copied to clipboard
| Challenge: | Automated Scientific Discovery (ASD) systems rely on parametric knowledge to generate and run code-based experiments. |
| Approach: | They propose a system that distills large collections of scientific Github repositories into a vetted library of working domain-specific code examples. |
| Outcome: | The proposed system produces more accurate, complete, and scientifically sound experiments than an agent with only general materials-science code examples. |
On the Challenges of Evaluating Compositional Explanations in Multi-Hop Inference: Relevance, Completeness, and Expert Ratings (2021.emnlp-main)
Copied to clipboard
| Challenge: | a large corpus of domain-expert relevance ratings augments a corpus for compositional explanations . a writer's study shows that the evaluations of compositional inference models underestimate performance . |
| Approach: | They construct a corpus of 126k domain-expert relevance ratings that augment explanations to standardized science exam questions. |
| Outcome: | The results show that evaluations underestimate performance of compositional explanations . they show that models regularly discover and produce valid explanations that are different than gold explanations. |
Multi-class Hierarchical Question Classification for Multiple Choice Science Exams (2020.lrec-1)
Copied to clipboard
Dongfang Xu, Peter Jansen, Jaycie Martin, Zhengnan Xie, Vikas Yadav, Harish Tayyar Madabushi, Oyvind Tafjord, Peter Clark
| Challenge: | Prior work has demonstrated that question classification (QC) can help answer a question more accurately. |
| Approach: | They propose to use a large dataset for question classification (QC) that contains 7,787 science exam questions paired with detailed classification labels from a fine-grained hierarchical taxonomy of 406 problem domains to train a BERT-based model. |
| Outcome: | The proposed model achieves a large (+0.12 MAP) gain while also achieving state-of-the-art performance on benchmark open-domain and biomedical QC datasets. |
Visually-Grounded Planning without Vision: Language Models Infer Detailed Plans from High-level Instructions (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Currently, the best performing systems can complete less than 1% of tasks successfully . we show that it is possible to generate gold multi-step plans from language directives alone without any visual input in 26% of unseen cases . |
| Approach: | They propose a task that requires a virtual robot to translate natural language directives into detailed sequences of actions that accomplish a goal in a real environment. |
| Outcome: | The proposed task generates gold multi-step plans from natural language directives in 26% of unseen cases. |
ScienceExamCER: A High-Density Fine-Grained Science-Domain Corpus for Common Entity Recognition (2020.lrec-1)
Copied to clipboard
| Challenge: | Named entity recognition identifies common classes of noun phrases in text, but these entity labels are sparse, limiting utility to downstream tasks. |
| Approach: | They propose a name-based named entity recognition model that annotates all content words with a fine-grained semantic class label. |
| Outcome: | The proposed model achieves 0.85 F1 on the science exam domain domain . the proposed model is a powerful tool for question answering and inference . |
Generating Literature-Driven Scientific Theories at Scale (2026.acl-long)
Copied to clipboard
| Challenge: | Contemporary automated scientific discovery systems focus on generating experiments, but higher-level activities such as theory building remain underexplored. |
| Approach: | They propose to synthesize theories from scientific literature using literature-grounding versus parametric knowledge. |
| Outcome: | The proposed method matches existing evidence better than parametric LLM memory generation. |
Behavior Cloned Transformers are Neurosymbolic Reasoners (2023.eacl-main)
Copied to clipboard
| Challenge: | In injecting actions from symbolic modules into the action space of a behavior cloned transformer agent increases performance on four text game benchmarks that test arithmetic, navigation, sorting, and common sense reasoning by an average of 22% . contemporary agents struggle on tasks such as navigation, admetic and other tasks that humans make use of external tools. |
| Approach: | They propose to inject actions from symbolic modules into the action space of a behavior cloned transformer agent to increase performance on four text game benchmarks that test arithmetic, navigation, sorting, and common sense reasoning by an average of 22% . |
| Outcome: | The proposed method improves performance on four text game benchmarks that test arithmetic, navigation, sorting, and common sense reasoning by 22%, allowing an agent to reach the highest possible performance on unseen games. |
Explaining Answers with Entailment Trees (2021.emnlp-main)
Copied to clipboard
Bhavana Dalvi, Peter Jansen, Oyvind Tafjord, Zhengnan Xie, Hannah Smith, Leighanna Pipatanangkura, Peter Clark
| Challenge: | ENTAILMENTBANK is the first dataset to contain multistep entailment trees. |
| Approach: | They propose to generate explanations in the form of entailment trees, a tree of multipremise entanglements steps from facts that are known to the hypothesis of interest. |
| Outcome: | The proposed model can generate explanations in the form of entailment trees . this is a tree of multipremise enttailment steps from facts known to the hypothesis of interest. |
TextWorldExpress: Simulating Text Games at One Million Steps Per Second (2023.eacl-demo)
Copied to clipboard
| Challenge: | Text-based games are limited in speed due to legacy tooling and limited implementations. |
| Approach: | They propose a text-game simulator that reimplements three common text game benchmarks and runs three orders of magnitude faster than their original implementations. |
| Outcome: | The proposed simulator increases throughput by three orders of magnitude, reaching over one million steps per second on common desktop hardware. |
Enhancing Systematic Decompositional Natural Language Inference Using Informal Logic (2024.emnlp-main)
Copied to clipboard
Nathaniel Weir, Kate Sanders, Orion Weller, Shreya Sharma, Dongwei Jiang, Zhengping Jiang, Bhavana Dalvi Mishra, Oyvind Tafjord, Peter Jansen, Peter Clark, Benjamin Van Durme
| Challenge: | Recent language models allow structured reasoning with text, but lack of a clear protocol for discerning entailment causes noisy datasets and limited performance gains. |
| Approach: | They propose a consistent approach to annotating decompositional entailment and evaluate its impact on LLM-based textual inference. |
| Outcome: | The proposed approach has higher internal consistency than prior decompositional entailment datasets and significantly improves proof quality and accuracy. |
ByteSized32: A Corpus and Challenge Task for Generating Task-Specific World Models Expressed as Text Games (2023.emnlp-main)
Copied to clipboard
| Challenge: | We show that language models can generate explicit, interpretable, and interactive world models of scientific and common-sense reasoning tasks. |
| Approach: | They propose a corpus of 32 reasoning-focused text games expressed as hundreds of lines of Python code to facilitate this task. |
| Outcome: | The proposed games can generate runnable games on unseen topics in 28% of cases. |
PDDLEGO: Iterative Planning in Textual Environments (2024.starsem-1)
Copied to clipboard
| Challenge: | Existing methods to plan in textual environments rely on a fully-observed environment where all entity states are known, but are not interpretable. |
| Approach: | They propose to use LLMs to generate a formal representation of the environment that can be solved by a symbolic planner. |
| Outcome: | The proposed model outperforms existing methods in the Coin Collector simulation and Cooking World simulations. |
WorldTree: A Corpus of Explanation Graphs for Elementary Science Questions supporting Multi-hop Inference (L18-1)
Copied to clipboard
| Challenge: | Existing methods of automated inference do not provide enough gold explanations to train models . standardized science exams are a challenge task for question answering . |
| Approach: | They propose to manually construct a corpus of explanations for standardized science exams . they also provide an explanation-centered tablestore that contains the knowledge to construct these explanations . |
| Outcome: | The proposed model provides detailed explanations for standardized science exams . the authors show that the proposed model can be trained on the basis of gold explanations . |
ScienceWorld: Is your Agent Smarter than a 5th Grader? (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing models cannot reason about or explain learned science concepts in novel contexts, despite transformer-based progress in question-answering and scientific text processing . |
| Approach: | They propose a benchmark to test agents’ scientific reasoning abilities in a new interactive text environment at the level of a standard elementary school science curriculum. |
| Outcome: | The proposed model outperforms a model trained for 100k steps in a standard elementary school science curriculum. |
Extracting Common Inference Patterns from Semi-Structured Explanations (D19-60)
Copied to clipboard
| Challenge: | Multi-hop inference suffers from semantic drift, or the tendency for chains of reasoning to "drift"' to unrelated topics. |
| Approach: | They propose to extract large high-confidence multi-hop inference patterns from a corpus of explanations by abstracting large-scale structure from logical sentences. |
| Outcome: | The proposed method extracts large high-confidence multi-hop inference patterns from a “matter” subset of elementary science exam questions. |
Matter-of-Fact: A Benchmark for Verifying the Feasibility of Literature-Supported Claims in Materials Science (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing systems generate hypothesis, run experiments, analyze data, and write or review papers, but they are costly and impractical. |
| Approach: | They propose a challenge dataset for determining the feasibility of hypotheses framed as claims and a temporally-filtered claim verification task using backtesting to test the validity of claims. |
| Outcome: | The proposed model performs well on retrieval augmented generation and code generation while performing 50% of the task. |
WorldTree V2: A Corpus of Science-Domain Structured Explanations and Inference Patterns supporting Multi-Hop Inference (2020.lrec-1)
Copied to clipboard
Zhengnan Xie, Sebastian Thiem, Jaycie Martin, Elizabeth Wainwright, Steven Marmorstein, Peter Jansen
| Challenge: | Standardized science questions require combining an average of 6 facts and as many as 16 facts to answer and explain. |
| Approach: | They propose to combine an average of 6 facts and as many as 16 facts to produce an answer for complex questions. |
| Outcome: | The proposed model is based on a corpus of 5,114 standardized science exam questions . it uses multi-fact explanations that combine science knowledge and world knowledge . |
CodeScientist: End-to-End Semi-Automated Scientific Discovery with Code-based Experimentation (2025.findings-acl)
Copied to clipboard
Peter Jansen, Oyvind Tafjord, Marissa Radensky, Pao Siangliulue, Tom Hope, Bhavana Dalvi Mishra, Bodhisattwa Prasad Majumder, Daniel S Weld, Peter Clark
| Challenge: | Automated scientific discovery (ASD) systems are limited in their evaluation of software artifacts and large volumes of research artifs are typically evaluated using conference-style paper review with limited evaluation of code. |
| Approach: | They propose a novel ASD system that frames ideation and experiment construction as a form of genetic search jointly over combinations of research articles and codeblocks defining common actions in a domain. |
| Outcome: | The proposed system returns 19 discoveries on machine-generated ideas in the domain of agents and virtual environments. |
TextGraphs 2019 Shared Task on Multi-Hop Inference for Explanation Regeneration (D19-53)
Copied to clipboard
| Challenge: | Detailed extended analyses of all submitted systems showed large relative improvements in accessing the most challenging multi-hop inference problems, while absolute performance remains low. |
| Approach: | The Shared Task on Multi-Hop Inference for Explanation Regeneration asks participants to regenerate detailed gold explanations for elementary science questions by selecting facts from a knowledge base of semi-structured tables. |
| Outcome: | The top-performing system achieved a mean average precision of 0.56 . the task combines facts from a knowledge base and supervised training data . |
CoSaTa: A Constraint Satisfaction Solver and Interpreted Language for Semi-Structured Tables of Sentences (2020.emnlp-demos)
Copied to clipboard
| Challenge: | Existing methods for inference using multiple facts are unable to combine more than two together. |
| Approach: | They present an intuitive constraint satisfaction solver and interpreted language for knowledge bases of semi-structured tables expressed as text. |
| Outcome: | The solver and interpreter are released as open source . the solver can express complex compositional "inference patterns" |
Can Language Models Serve as Text-Based World Simulators? (2024.acl-short)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have pointed towards an alternative approach by leveraging the huge amount of knowledge contained in their pre-training datasets. |
| Approach: | They build and use a benchmark to quantify how well text-based simulators can serve as text-driven world simulators. |
| Outcome: | The proposed benchmark aims to quantify how well language models can serve as world simulators. |
Extracting Space Situational Awareness Events from News Text (2022.lrec-1)
Copied to clipboard
Zhengnan Xie, Alice Saebom Kwak, Enfa George, Laura W. Dozal, Hoang Van, Moriba Jah, Roberto Furfaro, Peter Jansen
| Challenge: | Space situational awareness is the decisionmaking knowledge required to predict, avoid, operate through, or recover from the loss, disruption, or degradation of space services, capabilities, or activities. |
| Approach: | They construct a corpus of 48.5k news articles spanning all known active satellites between 2009 and 2020 that are annotated by humans with 15.9k labels for event slots. |
| Outcome: | The proposed system achieves an overall F1 between 53 and 91 per slot for event extraction in this low-resource, high-impact domain. |
From Words to Wires: Generating Functioning Electronic Devices from Natural Language Descriptions (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work shows that language models have a capacity for electronic circuit design from high-level textual descriptions, akin to code generation. |
| Approach: | They propose two benchmarks to assess a model's ability to design common microcontroller circuits and code in the Arduino ecosystem. |
| Outcome: | The proposed model knowledge of electrical components and the model's ability to generate full circuits and code in the Arduino ecosystem are evaluated. |