Papers by Jacob Andreas
A Probabilistic Framework for LLM Hallucination Detection via Belief Tree Propagation (2025.naacl-long)
Copied to clipboard
| Challenge: | Current large language models (LLMs) produce factually incorrect statements . |
| Approach: | They propose a probabilistic framework for LLM hallucination detection that generates a belief tree by expanding a statement into logically related claims and reasoning globally about the relationships between these claims. |
| Outcome: | The proposed method improves on multiple hallucination detection benchmarks by 3%-9% over state-of-the-art models. |
Grokking of Hierarchical Structure in Vanilla Transformers (2023.acl-short)
Copied to clipboard
| Challenge: | a recent study has shown that neural sequence models like transformers can generalize hierarchically when training for extended periods. |
| Approach: | They show that transformers can learn to generalize hierarchically after long training periods . they call this phenomenon structural grokking, which exhibits inverted U-shaped scaling in model depth . |
| Outcome: | The proposed model generalizes better than both very deep and very shallow models on multiple datasets. |
Task-Oriented Dialogue as Dataflow Synthesis (2020.tacl-1)
Copied to clipboard
Jacob Andreas, John Bufe, David Burkett, Charles Chen, Josh Clausman, Jean Crawford, Kate Crim, Jordan DeLoach, Leah Dorner, Jason Eisner, Hao Fang, Alan Guo, David Hall, Kristin Hayes, Kellie Hill, Diana Ho, Wendy Iwaszuk, Smriti Jha, Dan Klein, Jayant Krishnamurthy, Theo Lanman, Percy Liang, Christopher H. Lin, Ilya Lintsbakh, Andy McGovern, Aleksandr Nisnevich, Adam Pauls, Dmitrij Petters, Brent Read, Dan Roth, Subhro Roy, Jesse Rusak, Beth Short, Div Slomin, Ben Snyder, Stephon Striplin, Yu Su, Zachary Tellman, Sam Thomson, Andrei Vorobev, Izabela Witoszko, Jason Wolfe, Abby Wray, Yuchen Zhang, Alexander Zotov
| Challenge: | Existing approaches to task-oriented dialogue represent dialogue state as a dataflow graph . microsoft's SMCalFlow dataset features complex dialogues about events, weather, places, and people . |
| Approach: | They propose a dataflow graph-based dialogue agent that maps each user utterance to a program that extends this graph. |
| Outcome: | The proposed framework improves representability and predictability in natural dialogues . it uses dataflow graphs and metacomputation to map user intents to a program . |
Language-to-Code Translation with a Single Labeled Example (2024.emnlp-main)
Copied to clipboard
Kaj Bostrom, Harsh Jhamtani, Hao Fang, Sam Thomson, Richard Shin, Patrick Xia, Benjamin Van Durme, Jason Eisner, Jacob Andreas
| Challenge: | In-Context Inverse Programming (ICIP) bootstraps a language-to-code system using mostly unlabeled programs written using a potentially unfamiliar library or API. |
| Approach: | They propose a method for bootstrapping a language-to-code system using mostly unlabeled programs written using a potentially unfamiliar library or API. |
| Outcome: | The proposed method outperforms baselines across nine domains and 100 examples in a “nearly unsupervised” setting. |
Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks (2024.naacl-long)
Copied to clipboard
Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, Yoon Kim
| Challenge: | Recent language models possess impressive performance across a wide range of tasks . however, they often rely on narrow, non-transferable procedures for task-solving . |
| Approach: | They propose to evaluate language models using "counterfactual" task variants that deviate from standard tasks. |
| Outcome: | The proposed framework shows that language models perform better on a wide range of tasks compared to the default conditions. |
Language Modeling with Latent Situations (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches for improving coherence of language models are lacking . effective generation of coherent text requires reasoning about the world that text describes. |
| Approach: | They propose a family of approaches for improving coherence in language models . they use entity states as auxiliary supervision to improve LMs' reasoning ability . |
| Outcome: | The proposed method reduces errors by up to 16% by fine-tuning and prompting . it can be applied to LMs that are pre-trained to explicitly The suitcase is open . |
LexSym: Compositionality as Lexical Symmetry (2023.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to generalize compositional models fail to generalise from small datasets. |
| Approach: | They propose a domain-general and model-agnostic formulation of compositionality as a constraint on symmetries of data distributions rather than models. |
| Outcome: | The proposed procedure matches or surpasses state-of-the-art, task-specific models on COGS semantic parsing, SCAN and Alchemy instruction following, and CLEVR-CoGenT visual question answering datasets. |
What Context Features Can Transformer Language Models Use? (2021.acl-long)
Copied to clipboard
| Challenge: | Recent studies show that transformer-based language models benefit from conditioning on contexts of hundreds to thousands of previous tokens. |
| Approach: | They propose to use lexical and structural information to ablate usable information in transformer language models. |
| Outcome: | The proposed model improves when conditioning on contexts of thousands of previous tokens. |
The Whole Truth and Nothing But the Truth: Faithful and Controllable Dialogue Response Generation with Dataflow Transduction and Constrained Decoding (2023.findings-acl)
Copied to clipboard
Hao Fang, Anusha Balakrishnan, Harsh Jhamtani, John Bufe, Jean Crawford, Jayant Krishnamurthy, Adam Pauls, Jason Eisner, Jacob Andreas, Dan Klein
| Challenge: | In a task-oriented dialogue system, response generation is a conditional language model, but effective dialogue agents must balance fluent generation with stricter constraints. |
| Approach: | They propose a rule-based content selection model that transduces a dialogue agent’s actions and their results into context-free grammars representing the space of contextually acceptable responses. |
| Outcome: | The proposed architecture outperforms both rule-based and learned approaches in human evaluations of fluency, relevance, and truthfulness. |
Compositional Generalization for Neural Semantic Parsing via Span-level Supervised Attention (2021.naacl-main)
Copied to clipboard
Pengcheng Yin, Hao Fang, Graham Neubig, Adam Pauls, Emmanouil Antonios Platanios, Yu Su, Sam Thomson, Jacob Andreas
| Challenge: | Existing approaches to compositional generalization in semantic parsers focus on word-level alignments, but they focus on spans. |
| Approach: | They propose a span-level supervised attention loss that improves compositional generalization in semantic parsers by focusing on spans. |
| Outcome: | The proposed method improves on three benchmarks of compositional generalization. |
Value-Agnostic Conversational Semantic Parsing (2021.acl-long)
Copied to clipboard
Emmanouil Antonios Platanios, Adam Pauls, Subhro Roy, Yuchen Zhang, Alexander Kyte, Alan Guo, Sam Thomson, Jayant Krishnamurthy, Jason Wolfe, Jacob Andreas, Dan Klein
| Challenge: | Existing models rely on rich representations of dialogue history that include all previously generated components of the output. |
| Approach: | They propose a model that abstracts over values to focus prediction on type- and function-level context. |
| Outcome: | The proposed model outperforms baseline models by 7.3% and 10.6% on SMCalFlow and TreeDST datasets. |
Experience Grounds Language (2020.emnlp-main)
Copied to clipboard
Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, Nicolas Pinto, Joseph Turian
| Challenge: | aaron carroll: language understanding research is held back by a failure to relate language to the physical world it describes and to social interactions it facilitates. carroll says successful linguistic communication relies on a shared experience of the world. |
| Approach: | They propose to use a broader physical and social context to address communication problems . they argue that the current success of representation learning approaches is limited . |
| Outcome: | a new study suggests that the current success of representation learning requires a parallel tradition of research on the broader physical and social context of language to address the deeper questions of communication. |
Interpreting User Requests in the Context of Natural Language Standing Instructions (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing approaches to LLM-based dialogue modeling provide additional context for users to make requests. |
| Approach: | They propose an approach to LLM-based dialogue modeling where persistent user constraints and preferences are provided as additional context for such interfaces. |
| Outcome: | The proposed model achieves a maximum of 46% exact match on the prediction of 2.4K English dialogues with a language-to-program dataset. |
LLM Agents for Coordinating Multi-User Information Gathering (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent large language models (LLMs) are becoming a crucial building block in developing automated agents that can assist human users with complex tasks. |
| Approach: | They introduce PeopleJoin, a benchmark for evaluating LM-mediated collaborative problem solving. |
| Outcome: | The proposed benchmarks are adapted from existing benchmarks for database question answering and multi-document summarization. |
Quantifying Adaptability in Pre-trained Language Models with 500 Tasks (2022.naacl-main)
Copied to clipboard
| Challenge: | a recent study examines the features and limits of LM adaptability to new tasks . many questions about the nature and limits remain unanswered . |
| Approach: | They evaluate adaptability to new tasks using a new benchmark, TaskBench500 . they find adaptation procedures differ dramatically in their ability to memorize small datasets . |
| Outcome: | The proposed benchmark compares 500 procedurally generated sequence modeling tasks to a new benchmark. |
MisinfoEval: Generative AI in the Era of “Alternative Facts” (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing efforts to address misinformation on social media platforms are hampered by user biases and scalability challenges. |
| Approach: | They propose a framework for generating and comprehensively evaluating large language model based misinformation interventions using a simulated social media environment and personalized explanations tailored to users' beliefs. |
| Outcome: | The proposed framework improves accuracy at reliability labeling by up to 41.72% and personalized explanations appeal to users' pre-existing values. |
Toward In-Context Teaching: Adapting Examples to Students’ Misconceptions (2024.acl-long)
Copied to clipboard
| Challenge: | Using computational models as pedagogical tools is becoming increasingly popular, but how effective can these models adapt as teachers to students of different types? |
| Approach: | They propose a suite of models and evaluation methods that combine Bayesian student models and AToM to evaluate adaptive teaching methods. |
| Outcome: | The proposed models outperform LLM-based and standard Bayesian teaching methods in the evaluation of simulated students across three learning domains. |
Language Models as Agent Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Language models (LMs) are trained on collections of documents written by individual human agents to achieve specific goals in the outside world. |
| Approach: | a new study shows that language models are models of communicative intentions in a specific, narrow sense . despite recent progress, today's language models still make odd predictions and conspicuous errors . |
| Outcome: | a survey of LMs shows that they can model communicative intentions in a specific, narrow sense . despite recent progress, current models still make odd predictions and conspicuous errors . |
Lexicon Learning for Few Shot Sequence Modeling (2021.acl-long)
Copied to clipboard
| Challenge: | Past work has shown that many failures of systematic generalization arise from neural models’ inability to disentangle lexical phenomena from syntactic ones. |
| Approach: | They propose a lexical translation mechanism that generalizes existing copy mechanisms to incorporate learned, decontextualized, token-level translation rules. |
| Outcome: | The proposed model improves generalization on a diverse set of sequence modeling tasks drawn from cognitive science, formal semantics, and machine translation. |
Good-Enough Compositional Data Augmentation (2020.acl-main)
Copied to clipboard
| Challenge: | a proposed data augmentation protocol provides a compositional inductive bias in conditional and unconditional sequence models. |
| Approach: | They propose a data augmentation protocol that provides a compositional inductive bias in conditional and unconditional sequence models by replacing discontinuous fragments with other fragments that appear in at least one similar environment. |
| Outcome: | The proposed protocol reduces error rate by 87% on diagnostic tasks and 16% on semantic parsing tasks. |
Language Modeling with Editable External Knowledge (2025.findings-naacl)
Copied to clipboard
| Challenge: | Continual learning methods tackle the problem of a changing world by incrementally training on new information. |
| Approach: | They propose a retrieval-augmented generation approach that incrementally deletes or rewrites other entries in the knowledge base each time a document is added. |
| Outcome: | The proposed model improves accuracy relative to conventional retrieval-augmented generation by 7-13% and 6-10% absolute. |
Implicit Representations of Meaning in Neural Language Models (2021.acl-long)
Copied to clipboard
| Challenge: | Neural language models (NLMs) encode lexical relations and syntactic structure, but their effectiveness is still unclear. |
| Approach: | They propose to use text as a model to model entities and situations as they evolve throughout a discourse. |
| Outcome: | The proposed models have functional similarities to linguistic models of dynamic semantics and can be learned with only text as training data. |
Regularized Conventions: Equilibrium Computation as a Model of Pragmatic Reasoning (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing computational models of pragmatics are implemented as recursive reasoning procedures, in which listeners interpret utterances by reasoning about the intentions of less-sophisticated speakers. |
| Approach: | They propose a game-theoretic model of pragmatics that combines a recursive reasoning model with a regularized conventions model to obtain principled pragmatic language generation procedures. |
| Outcome: | The proposed model matches or improves predictions made by iterated best response and Rational Speech Acts models of language understanding. |
Pragmatically Informative Text Generation (N19-1)
Copied to clipboard
| Challenge: | Existing approaches to pragmatics have been used to improve the informativeness of generated text in grounded language learning problems. |
| Approach: | They propose to use pragmatics to improve the informativeness of conditional text models . they propose to apply pragmatic reasoning to more traditional language generation tasks . |
| Outcome: | The proposed methods improve the performance of strong existing systems for abstractive summarization and generation from structured meaning representations. |
Unified Pragmatic Models for Generating and Following Instructions (N18-1)
Copied to clipboard
| Challenge: | a new technique for layering explicit pragmatic inference on top of models for sequential tasks is proposed . explicit pragmatic reasoning is used to generate and follow natural language instructions . |
| Approach: | They propose a pragmatic speaker that uses the base listener to simulate the interpretation of candidate descriptions and a listener that reasons counterfactually about alternative descriptions. |
| Outcome: | The proposed model improves state-of-the-art models for interpreting human instructions and speaker models in diverse settings. |
Deductive Closure Training of Language Models for Coherence, Accuracy, and Updatability (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing language models (LMs) generate factually correct text and estimate truth values of individual claims, but they do not reflect a coherent, manipulable model of the world. |
| Approach: | They propose a method that uses language models to identify implications of (and contradictions within) the text they generate. |
| Outcome: | The proposed method improves LM factuality by 3-26% across the CREAK, MQuAKE, and Reversal Curse datasets. |
Pushdown Layers: Encoding Recursive Structure in Transformer Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pushdown Layers model recursive state via stack tape that tracks estimated depths of tokens in incremental parsing . pushdown layers are drop-in replacement for standard self-attention . recursion is a key component of many aspects of intelligent behavior, authors say . |
| Approach: | They propose a self-attention layer that models recursive state via a stack tape . Pushdown Layers is a drop-in replacement for standard self- attention . |
| Outcome: | The proposed self-attention layer improves on parse tasks with a recursive-state model . it can model recursion using a stack tape that tracks estimated depths of tokens . |
Skill Induction and Planning with Latent Language (2022.acl-long)
Copied to clipboard
| Challenge: | a framework for learning hierarchical policies from demonstrations is lacking . we use sparse annotations to guide the discovery of reusable skills . |
| Approach: | They propose a framework for learning hierarchical policies from demonstrations using sparse annotations. |
| Outcome: | The proposed model outperforms other models with access to ground-truth plans in the ALFRED simulation environment. |
Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness? (2023.emnlp-main)
Copied to clipboard
| Challenge: | Past work has found that query–probe disagreements occur in situations where either probes or queries are uncertain. |
| Approach: | They propose to identify three different classes of disagreement, which they term confabulation, deception, and heterogeneity. |
| Outcome: | The proposed methods are more accurate than LM outputs and better on uncertain answers than queries and probes. |
Hierarchical Phrase-Based Sequence-to-Sequence Learning (2022.emnlp-main)
Copied to clipboard
| Challenge: | a neural transducer that incorporates hierarchical phrases as a source of inductive bias during training and as explicit constraints during inference is described. |
| Approach: | They propose a neural transducer that incorporates hierarchical phrases as a source of inductive bias during training and as explicit constraints during inference. |
| Outcome: | The proposed model performs well on small scale machine translation benchmarks. |
Multitasking Inhibits Semantic Drift (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing studies have found that LLP training is prone to semantic drift (use of messages inconsistent with their natural language meanings) |
| Approach: | They propose to use latent language policies to train neural LLPs to eliminate semantic drift in a well-studied family of signaling games to reduce drift and improve sample efficiency. |
| Outcome: | The proposed model eliminates semantic drift in a well-studied family of signaling games while improving sample efficiency. |
How Do Neural Sequence Models Generalize? Local and Global Cues for Out-of-Distribution Prediction (2021.emnlp-main)
Copied to clipboard
| Challenge: | Using RNN and transformer language models, we show consistent generalization in out-of-distribution contexts. |
| Approach: | They propose two idealized models of generalization in next-word prediction . they show that neural language models interpolate between these two forms of generalisation . |
| Outcome: | The proposed models exhibit consistent generalization in out-of-distribution contexts. |
Visual Grounding Helps Learn Word Meanings in Low-Data Regimes (2024.naacl-long)
Copied to clipboard
| Challenge: | Modern neural language models (LMs) require distinctly un-human-like ways to achieve these results. |
| Approach: | They train a diverse set of LM architectures with and without auxiliary visual supervision on datasets of varying scales. |
| Outcome: | The proposed models exhibit better learning of syntactic categories, lexical relations, semantic features, word similarity and alignment with human neural representations. |
Lexicon-Level Contrastive Visual-Grounding Improves Language Modeling (2024.findings-acl)
Copied to clipboard
| Challenge: | Neural language models (LMs) are trained on orders of magnitude more language data than human language learners receive, but without supervision from other sensory modalities that play a crucial role in human learning. |
| Approach: | They propose a grounded language learning procedure that leverages visual supervision to improve textual representations. |
| Outcome: | The proposed procedure outperforms standard language-only models in terms of learning efficiency in small and developmentally plausible data regimes and improves perplexity by around 5% on multiple language modeling tasks compared to other models trained on the same amount of text data. |
Guided K-best Selection for Semantic Parsing Annotation (2022.acl-demo)
Copied to clipboard
Anton Belyy, Chieh-yang Huang, Jacob Andreas, Emmanouil Antonios Platanios, Sam Thomson, Richard Shin, Subhro Roy, Aleksandr Nisnevich, Charles Chen, Benjamin Van Durme
| Challenge: | a prototype model trained on a small amount of data is not available, leading to limited prediction performance. |
| Approach: | They propose a human-in-the-loop process that generates a set of valid candidates and allows users to quickly traverse the set and filter incorrect parses. |
| Outcome: | The proposed process can be used to efficiently traverse the candidate set and select the correct parse, with minimal modification when necessary. |
Learning with Latent Language (N18-1)
Copied to clipboard
| Challenge: | Using the space of natural language strings as a parameter space is an effective way to capture natural task structure. |
| Approach: | They propose to use natural language as a parameter space for few-shot learning problems including classification, transduction and policy search. |
| Outcome: | The proposed model outperforms models with a linguistic parameterization on image classification, text editing, and reinforcement learning. |
Towards Tracing Knowledge in Language Models Back to the Training Data (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Prior work on training data attribution (TDA) may offer effective tools for identifying such examples, known as "proponents". |
| Approach: | They propose a benchmark to identify which training examples taught an LM to generate a particular factual assertion. |
| Outcome: | The proposed methods have lower proponent-retrieval precision than baselines that do not have access to the LM. |