Papers by Dan Klein

PREADD: Prefix-Adaptive Decoding for Controlled Text Generation (2023.findings-acl)

Challenge:	Using electromyography, we can convert silently mouthed words into audible speech . prior work focused on training speech synthesis models from vocalized data .
Approach:	They propose a method of training on silent EMG by transferring audio targets from vocalized to silent signals and propose voicing task using muscle sensor measurements.
Outcome:	The proposed method greatly improves intelligibility of audio generated from silent EMG compared to baseline that only trains with vocalized data.

Copied to clipboard

Jonathan Pei, Kevin Yang, Dan Klein

From SPMRL to NMRL: What Did We Learn (and Unlearn) in a Decade of Parsing Morphologically-Rich Languages (MRLs)? (2020.acl-main)

Challenge:	Existing methods for controlling text generation rely on auxiliary expert models but require training data.
Approach:	They propose a flexible method for controlled text generation that uses a prefix-prepended prompt instead of an external model.
Outcome:	The proposed method outperforms auxiliary-expert methods on three tasks by 12% or more on main metrics for each task.

Copied to clipboard

Reut Tsarfaty, Dan Bareket, Stav Klein, Amit Seker

Multilingual Constituency Parsing with Self-Attention and Pre-Training (P19-1)

Challenge:	a decade has passed since the establishment of SPMRL to address the peculiar challenges of Statistical Parsing for Morphologically-rich languages (MRLs).
Approach:	They propose a framework for parsing MRLs and propose implementing symbolic ideas into modern neural architectures.
Outcome:	The proposed strategies are based on the multi-tagging task in Hebrew, a morphologically-rich, high-fusion, language.

Copied to clipboard

Nikita Kitaev, Steven Cao, Dan Klein

Modular Networks for Compositional Instruction Following (2021.naacl-main)

Challenge:	a range of pre-training conditions can be used for constituency parsing, but large model sizes make it expensive to train separate models for each language.
Approach:	They compare the benefits of no pre-training, fastText, ELMo, and BERT for English . they also find that pre- training is beneficial across all 11 languages tested .
Outcome:	The proposed model outperforms fastText, ELMo, and BERT for English . but large model sizes make it expensive to train separate models for each language .

Copied to clipboard

Rodolfo Corona, Daniel Fried, Coline Devin, Dan Klein, Trevor Darrell

Challenge:	Standard instruction following models struggle on novel compositions of subgoals observed during training.
Approach:	They propose a modular architecture that follows natural language instructions that describe sequences of diverse subgoals.
Outcome:	The proposed architecture improves generalization to novel subgoals and environments unseen in training.

Task-Oriented Dialogue as Dataflow Synthesis (2020.tacl-1)

Copied to clipboard

Jacob Andreas, John Bufe, David Burkett, Charles Chen, Josh Clausman, Jean Crawford, Kate Crim, Jordan DeLoach, Leah Dorner, Jason Eisner, Hao Fang, Alan Guo, David Hall, Kristin Hayes, Kellie Hill, Diana Ho, Wendy Iwaszuk, Smriti Jha, Dan Klein, Jayant Krishnamurthy, Theo Lanman, Percy Liang, Christopher H. Lin, Ilya Lintsbakh, Andy McGovern, Aleksandr Nisnevich, Adam Pauls, Dmitrij Petters, Brent Read, Dan Roth, Subhro Roy, Jesse Rusak, Beth Short, Div Slomin, Ben Snyder, Stephon Striplin, Yu Su, Zachary Tellman, Sam Thomson, Andrei Vorobev, Izabela Witoszko, Jason Wolfe, Abby Wray, Yuchen Zhang, Alexander Zotov

QANom: Question-Answer driven SRL for Nominalizations (2020.coling-main)

Challenge:	Existing approaches to task-oriented dialogue represent dialogue state as a dataflow graph . microsoft's SMCalFlow dataset features complex dialogues about events, weather, places, and people .
Approach:	They propose a dataflow graph-based dialogue agent that maps each user utterance to a program that extends this graph.
Outcome:	The proposed framework improves representability and predictability in natural dialogues . it uses dataflow graphs and metacomputation to map user intents to a program .

Copied to clipboard

Ayal Klein, Jonathan Mamou, Valentina Pyatkin, Daniela Stepanov, Hangfeng He, Dan Roth, Luke Zettlemoyer, Ido Dagan

Natural Language Interfaces

Challenge:	Traditionally, SRL annotations focus on verbal predicates, but other types of predicate are frequent in natural language.
Approach:	They propose a semantic scheme for capturing predicate-argument relations for nominalizations, termed QANom, using crowdsourcing and QA-driven annotations.
Outcome:	The proposed scheme outperforms existing annotations and is useful for downstream tasks.

The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels (2024.naacl-long)

Copied to clipboard

Eve Fleisig, Su Lin Blodgett, Dan Klein, Zeerak Talat

An Improved Model for Voicing Silent Speech (2021.acl-short)

Challenge:	a line of recent work has illustrated that annotators disagree for many reasons . capturing disagreements can improve model performance and calibration, authors argue .
Approach:	They propose a new paradigm shift in data labeling for machine learning that challenges annotator disagreement by treating disagreement as a valuable source of information.
Outcome:	The proposed approaches challenge annotator disagreement and provide recommendations for the data labeling pipeline and avenues for future research.

Copied to clipboard

David Gaddy, Dan Klein

Modular Visual Question Answering via Code Generation (2023.acl-short)

Challenge:	Existing models for voicing silent speech use hand-designed features instead of EMG signals.
Approach:	They propose to use facial electromyography signals as input instead of hand-designed features to give the model greater flexibility to learn its own features.
Outcome:	The proposed model improves state-of-the-art on an open vocabulary intelligibility evaluation by 25.8%.

Copied to clipboard

Sanjay Subramanian, Medhini Narasimhan, Kushal Khangaonkar, Kevin Yang, Arsha Nagrani, Cordelia Schmid, Andy Zeng, Trevor Darrell, Dan Klein

The Whole Truth and Nothing But the Truth: Faithful and Controllable Dialogue Response Generation with Dataflow Transduction and Constrained Decoding (2023.findings-acl)

Challenge:	a framework for visual question answering is based on modular code generation . the scope of reasoning needed for visual questions is vast, and requires many skills .
Approach:	They propose a framework that formulates visual question answering as modular code generation.
Outcome:	The proposed framework improves accuracy on COVR and GQA datasets by 3% and 2% compared to the few-shot baseline that does not employ code generation.

Copied to clipboard

Hao Fang, Anusha Balakrishnan, Harsh Jhamtani, John Bufe, Jean Crawford, Jayant Krishnamurthy, Adam Pauls, Jason Eisner, Jacob Andreas, Dan Klein

Dialogue Systems & Conversational Agents

Challenge:	In a task-oriented dialogue system, response generation is a conditional language model, but effective dialogue agents must balance fluent generation with stricter constraints.
Approach:	They propose a rule-based content selection model that transduces a dialogue agent’s actions and their results into context-free grammars representing the space of contextually acceptable responses.
Outcome:	The proposed architecture outperforms both rule-based and learned approaches in human evaluations of fluency, relevance, and truthfulness.

Reference-Centric Models for Grounded Collaborative Dialogue (2021.emnlp-main)

Copied to clipboard

Daniel Fried, Justin Chiu, Dan Klein

Dialogue Systems & Conversational Agents

Challenge:	Using a structured referent grounding module, we can effectively ground and inform a partner's utterances to their own context.
Approach:	They propose a grounded neural dialogue model that works with people in a partially-observable reference game.
Outcome:	The proposed model outperforms state-of-the-art models on a spatial grounding dialogue task and achieves a 20% relative improvement in human evaluations.

Learned Incremental Representations for Parsing (2022.acl-long)

Copied to clipboard

Nikita Kitaev, Thomas Lu, Dan Klein

Representation Learning

Challenge:	a new syntactic representation that commits to syntakic choices is proposed for humans . we use a system that uses only incremental processing of a prefix to predict the word in a sentence .
Approach:	They propose a syntactic representation that commits to syntakic choices incrementally . they say the system can achieve 93.72 F1 on the Penn Treebank with as few as 5 bits per word .
Outcome:	The proposed representation achieves 93.72 F1 on the Penn Treebank with as few as 5 bits per word . the analysis of the representations shows they have interpretable features and deferred resolution of syntactic ambiguities.

Decomposing Complex Queries for Tip-of-the-tongue Retrieval (2023.findings-emnlp)

Copied to clipboard

Kevin Lin, Kyle Lo, Joseph Gonzalez, Dan Klein

Information Retrieval

Challenge:	Tip-of-the-tongue retrieval is a retrieval setting in which a user is unable to formulate a precise query that identifies a sought item . a framework that decomposes complex queries into subqueries can improve gold book recall .
Approach:	They propose a framework for handling tip-of-the-tongue queries by decomposing queries into individual clues routing them to specialized retrievers.
Outcome:	The proposed framework improves gold book recall up to 6% on a new query-book pair . it takes advantage of off-the-shelf retrievers or incorporates retriever-specific logic .

Semantic Evaluation for Text-to-SQL with Distilled Test Suites (2020.emnlp-main)

Copied to clipboard

Ruiqi Zhong, Tao Yu, Dan Klein

Programming Languages in NLP

Challenge:	Existing methods to evaluate semantic accuracy of Text-to-SQL models are not accurate.
Approach:	They propose a test suite accuracy method to approximate semantic accuracy for Text-to-SQL models.
Outcome:	The proposed method evaluates 21 models submitted to the Spider leader board and manually checks on 100 examples.

Which One? Leveraging Context Between Objects and Multiple Views for Language Grounding (2024.naacl-long)

Copied to clipboard

Chancharik Mitra, Abrar Anwar, Rodolfo Corona, Dan Klein, Trevor Darrell, Jesse Thomason

Reasoning

Challenge:	Existing methods for identifying object referents of language expressions consider target and distractor objects independently and pool multiple views before grounding.
Approach:	They propose a model that selects an object referent based on language that distinguishes between two similar objects and a multi-view approach to grounding in context model which reduces the relative error by 12.9% .
Outcome:	The proposed model improves on the SNARE object reference task with a relative error reduction of 12.9% and an absolute improvement of 2.7%.

Detoxifying Language Models Risks Marginalizing Minority Voices (2021.naacl-main)

Copied to clipboard

Albert Xu, Eshaan Pathak, Eric Wallace, Suchin Gururangan, Maarten Sap, Dan Klein

Value-Agnostic Conversational Semantic Parsing (2021.acl-long)

Challenge:	Existing detoxification techniques have been proposed to mitigate toxic LM generations . e.g., detoxification makes LMs more brittle to distribution shift, especially on language used by marginalized groups .
Approach:	They propose to use detoxification techniques to reduce toxic LM generations without affecting perplexity or generation quality on nontoxic inputs.
Outcome:	The proposed methods hurt equity on language used by marginalized groups, the authors show . they show that detoxification makes LMs more brittle to distribution shift, they say .

Copied to clipboard

Emmanouil Antonios Platanios, Adam Pauls, Subhro Roy, Yuchen Zhang, Alexander Kyte, Alan Guo, Sam Thomson, Jayant Krishnamurthy, Jason Wolfe, Jacob Andreas, Dan Klein

Natural Language Interfaces

Challenge:	Existing models rely on rich representations of dialogue history that include all previously generated components of the output.
Approach:	They propose a model that abstracts over values to focus prediction on type- and function-level context.
Outcome:	The proposed model outperforms baseline models by 7.3% and 10.6% on SMCalFlow and TreeDST datasets.

Are Larger Pretrained Language Models Uniformly Better? Comparing Performance at the Instance Level (2021.findings-acl)

Copied to clipboard

Ruiqi Zhong, Dhruba Ghosh, Dan Klein, Jacob Steinhardt

Incorporating Worker Perspectives into MTurk Annotation Practices for NLP (2023.emnlp-main)

Challenge:	Larger models have higher out-of-distribution robustness, while smaller models have lower accuracy on rare subgroups.
Approach:	They develop statistically rigorous methods to investigate whether large models are better on every instance . they find that individual predictions are highly sensitive to noise in the randomness in training .
Outcome:	The proposed model is worse than BERT-MINI on 1-4% of instances across MNLI, SST-2, and QQP, compared to the overall accuracy improvement of 2-10%.

Copied to clipboard

Olivia Huang, Eve Fleisig, Dan Klein

Tetra-Tagging: Word-Synchronous Parsing with Linear-Time Inference (2020.acl-main)

Challenge:	Current approaches to data collection for natural language processing on Amazon Mechanical Turk (MTurk) are susceptible to issues regarding workers’ rights and poor response quality without considering the perspectives of MTurq workers.
Approach:	They conducted a critical literature review and a survey of MTurk workers to address open questions regarding fair payment, worker privacy, data quality, and considering worker incentives.
Outcome:	The findings suggest that future studies may better account for MTurk workers’ experiences in order to respect workers' rights and improve response quality.

Copied to clipboard

Nikita Kitaev, Dan Klein

ThoughtSculpt: Reasoning with Intermediate Revision and Search (2025.findings-naacl)

Challenge:	Using custom architectures, constituency parsers are limited and require specialized hardware.
Approach:	They propose an algorithm that assigns labels to each word in a sentence in parallel and then performs a reconciliation phase to extract a tree in (empirically) linear time.
Outcome:	The proposed model achieves 95.4 F1 on the WSJ test set while also achieving substantial speedups compared to current state-of-the-art parsers with comparable accuracies.

Copied to clipboard

Yizhou Chi, Kevin Yang, Dan Klein

Reasoning

Challenge:	THOUGHTSCULPT is a general reasoning and search method for tasks with outputs that can be decomposed into components.
Approach:	They propose a general reasoning and search method for tasks with outputs that can be decomposed into components.
Outcome:	THOUGHTSCULPT outperforms state-of-the-art reasoning methods on three tasks . authors show that distinct prompting strategies can influence the performance of LLMs .

Improving Pacing in Long-Form Story Planning (2023.findings-emnlp)

Copied to clipboard

Yichen Wang, Kevin Yang, Xiaoming Liu, Dan Klein

Challenge:	Existing systems for writing long-form stories suffer from unnatural pacing, whether glossing over important events or over-elaborating on insignificant details.
Approach:	They propose a system that uses a concreteness evaluator to judge which of two events is more concrete.
Outcome:	The proposed system improves pacing when automatically generating story outlines.

Inferring Rewards from Language in Context (2022.acl-long)

Copied to clipboard

Jessy Lin, Daniel Fried, Dan Klein, Anca Dragan

Challenge:	a new study grounding language to reward functions extends the standard instruction following setup in this way.
Approach:	They propose a model that infers rewards from language pragmatically by reasoning about how speakers choose utterances to elicit desired actions and reveal information about their preferences.
Outcome:	The proposed model infers rewards from language pragmatically on a flight–booking task with natural language.

Re-evaluating the Need for Visual Signals in Unsupervised Grammar Induction (2024.findings-naacl)

Copied to clipboard

Boyi Li, Rodolfo Corona, Karttikeya Mangalam, Catherine Chen, Daniel Flaherty, Serge Belongie, Kilian Weinberger, Jitendra Malik, Trevor Darrell, Dan Klein

Are You Looking? Grounding to Multiple Modalities in Vision-and-Language Navigation (P19-1)

Challenge:	Recent studies show multimodal inputs can improve grammar induction, but weak textual baselines are needed for training.
Approach:	They use a fixed grammar family to compare multimodal grammar induction methods . they find multimodal inputs can improve grammar induction by grounding textual inputs to the visual world .
Outcome:	The proposed model outperforms weaker baselines on four benchmark datasets.

Copied to clipboard

Ronghang Hu, Daniel Fried, Anna Rohrbach, Dan Klein, Trevor Darrell, Kate Saenko

FactTrack: Time-Aware World State Tracking in Story Outlines (2025.naacl-long)

Challenge:	Existing models that ground language into visual appearance and route structure are outperforming their visual counterparts in unseen new environments.
Approach:	They propose to decompose the grounding procedure into a set of expert models with access to different modalities and ensemble them at prediction time.
Outcome:	The proposed model outperforms models with only route structure and visual features on the benchmark Room-to-Room dataset.

Copied to clipboard

Zhiheng Lyu, Kevin Yang, Lingpeng Kong, Dan Klein

Ghostbuster: Detecting Text Ghostwritten by Large Language Models (2024.naacl-long)

Challenge:	Existing language models still struggle to reason over long context windows . et al., 2022, show that long context generation is a challenge for LLMs .
Approach:	They propose a method for tracking atomic facts and addressing factual contradictions . they use a four-step pipeline to update a world state data structure for each new event .
Outcome:	The proposed method outperforms a baseline and fair method on story outlines.

Copied to clipboard

Vivek Verma, Eve Fleisig, Nicholas Tomlin, Dan Klein

When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Tasks (2023.emnlp-main)

Challenge:	Ghostbuster is a system that passes documents through weaker language models, runs a structured search over possible combinations of their features, and trains a classifier on the selected features.
Approach:	They propose a method that passes documents through weaker language models, runs a structured search over possible combinations of their features, and trains a classifier on the selected features.
Outcome:	The proposed method outperforms existing detectors and a new baseline on student essays, creative writing, and news articles.

Copied to clipboard

Eve Fleisig, Rediet Abebe, Dan Klein

Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections (2021.findings-emnlp)

Challenge:	a number of studies have questioned assumptions of majority vote aggregated labels.
Approach:	They construct a model that predicts individual annotator ratings on potentially offensive text and combines this information with the predicted target group of the text to predict the ratings of target group members.
Outcome:	The proposed model raises performance over baseline by 22% and 33% at predicting variance among annotators.

Copied to clipboard

Ruiqi Zhong, Kristy Lee, Zheng Zhang, Dan Klein

A Streaming Approach For Efficient Batched Beam Search (2020.emnlp-main)

Challenge:	Large pre-trained language models (LMs) have a surprising ability to perform zero-shot learning.
Approach:	They propose to fine-tune pre-trained language models to optimize the zero-shot learning objective by aggregating 43 existing datasets and annotating 441 label descriptions in a question-answering format.
Outcome:	The proposed model outperforms a same-sized QA model and the previous SOTA zero-shot learning system on unseen tasks.

Copied to clipboard

Kevin Yang, Violet Yao, John DeNero, Dan Klein

What Evidence Do Language Models Find Convincing? (2024.acl-long)

Challenge:	During decoding, candidates terminate or are pruned according to heuristics, a streaming method is used to "refill" the batch after it finishes translating some fraction of the current batch.
Approach:	They propose an efficient batching strategy for variable-length decoding on GPU architectures by streamlining the batching process.
Outcome:	The proposed method reduces runtime by 71% compared to a fixed-width beam search baseline and 17% compared with a variable-widness baseline while matching baselines’ BLEU.

Copied to clipboard

Alexander Wan, Eric Wallace, Dan Klein

Automated Crossword Solving (2022.acl-long)

Challenge:	Current retrieval-augmented language models are tasked with subjective, contentious, and conflicting queries.
Approach:	They construct a dataset that pairs controversial queries with real-world evidence documents . they find current models rely heavily on relevance of a website to the query .
Outcome:	The proposed dataset pairs controversial queries with real-world evidence documents that contain different facts, arguments, and answers.

Copied to clipboard

Eric Wallace, Nicholas Tomlin, Albert Xu, Kevin Yang, Eshaan Pathak, Matthew Ginsberg, Dan Klein

Question Answering

Challenge:	Using neural question answering models, our system generates answer candidates and then combines loopy belief propagation with local search to find full puzzle solutions.
Approach:	They propose a new approach to automatically solving crossword puzzles that uses neural question answering models and loopy belief propagation with local search to find full puzzle solutions.
Outcome:	The proposed system outperforms even the best human solvers and can solve crosswords from a wide range of domains with perfect accuracy.

Constrained Language Models Yield Few-Shot Semantic Parsers (2021.emnlp-main)

Copied to clipboard

Richard Shin, Christopher Lin, Sam Thomson, Charles Chen, Subhro Roy, Emmanouil Antonios Platanios, Adam Pauls, Dan Klein, Jason Eisner, Benjamin Van Durme

Are Layout-Infused Language Models Robust to Layout Distribution Shifts? A Case Study with Scientific Documents (2023.findings-acl)

Challenge:	Large pretrained language models excel at generating natural language, but they are not efficient for task specific semantic parsing.
Approach:	They propose to use large pretrained language models as few-shot semantic parsers . they paraphrase inputs into a controlled sublanguage resembling English .
Outcome:	The proposed model can generate surprisingly accurate models on multiple tasks with minimal code and data.

Copied to clipboard

Catherine Chen, Zejiang Shen, Dan Klein, Gabriel Stanovsky, Doug Downey, Kyle Lo

Semantic Scaffolds for Pseudocode-to-Code Generation (2020.acl-main)

Challenge:	Recent work has shown that infusing layout features into language models improves processing of visually-rich documents such as scientific papers.
Approach:	They propose a method to evaluate layout-infused language models that incorporate layout features into their models to emulate layout distribution shifts.
Outcome:	The proposed model performs better under layout distribution shifts than in-distribution conditions.

Copied to clipboard

Ruiqi Zhong, Mitchell Stern, Dan Klein

Programming Languages in NLP

Challenge:	Existing methods for program generation use lightweight structures to represent high-level semantics and syntactic composition of a program.
Approach:	They propose a method for program generation based on semantic scaffolds . they use line-level natural language pseudocode annotations to search for programs .
Outcome:	The proposed method achieves 10% improvement in top-100 accuracy over the current state-of-the-art method.

FUDGE: Controlled Text Generation With Future Discriminators (2021.naacl-main)

Copied to clipboard

Kevin Yang, Dan Klein

Text Generation

Challenge:	Recent advances in large pretrained language models allow us to generate increasingly realistic text by modeling a distribution P (X) over natural language sequences X.
Approach:	They propose a flexible and modular method for controlled text generation that uses a Bayesian decomposition of the conditional distribution of G given an attribute predictor and can easily compose predictors for multiple desired attributes.
Outcome:	The proposed method can be easily composed and performs three tasks.

Neural Unsupervised Reconstruction of Protolanguage Word Forms (2023.acl-long)

Copied to clipboard

Andre He, Nicholas Tomlin, Dan Klein

Constituency Parsing with a Self-Attentive Encoder (P18-1)

Challenge:	Existing methods for reconstructing ancient word forms use expectation-maximization . past work has used this method to predict simple phonological changes .
Approach:	They extend expectation-maximization to predict phonological changes between ancient word forms and their cognates in modern languages.
Outcome:	The proposed model reduces edit distance from the target word forms compared to previous methods.

Copied to clipboard

Nikita Kitaev, Dan Klein

Centering the Margins: Outlier-Based Identification of Harmed Populations in Toxicity Detection (2023.emnlp-main)

Challenge:	Recent work on LSTM encoders based on recurrent neural networks has led to improvements in constituency parsing accuracy.
Approach:	They propose to replace an LSTM encoder with a self-attentive architecture to improve a discriminative constituency parser.
Outcome:	The proposed model outperforms the previous best-published results on 8 of the 9 languages in the SPMRL dataset.

Copied to clipboard

Vyoma Raman, Eve Fleisig, Dan Klein

Pragmatically Informative Text Generation (N19-1)

Challenge:	toxicity detection models focus on marginalized groups, but they obscure harms faced by intersectional subgroups.
Approach:	They use outlier detection to identify text about people with demographic attributes distant from the "norm" they find model performance is worse for demographic outliers than non-outliers .
Outcome:	The proposed model performance is worse for outliers than non-outliers, the authors say . their analysis also shows that outlier analysis can identify harms faced by intersectional groups .

Copied to clipboard

Sheng Shen, Daniel Fried, Jacob Andreas, Dan Klein

Non-Programmers Can Label Programs Indirectly via Active Examples: A Case Study with Text-to-SQL (2023.emnlp-main)

Challenge:	Existing approaches to pragmatics have been used to improve the informativeness of generated text in grounded language learning problems.
Approach:	They propose to use pragmatics to improve the informativeness of conditional text models . they propose to apply pragmatic reasoning to more traditional language generation tasks .
Outcome:	The proposed methods improve the performance of strong existing systems for abstractive summarization and generation from structured meaning representations.

Copied to clipboard

Ruiqi Zhong, Charlie Snell, Dan Klein, Jason Eisner

Programming Languages in NLP

Challenge:	Using APEL, non-programmers can annotate natural language utterances with complex programs that represent their meaning.
Approach:	They introduce a framework in which non-programmers select among candidate programs generated by a seed semantic parser.
Outcome:	The proposed framework achieves the same annotation accuracy as the original expert annotators (75%) and exposes many subtle errors in the original annotations.

Unified Pragmatic Models for Generating and Following Instructions (N18-1)

Copied to clipboard

Daniel Fried, Jacob Andreas, Dan Klein

Explainability & Interpretability in NLP

Challenge:	a new technique for layering explicit pragmatic inference on top of models for sequential tasks is proposed . explicit pragmatic reasoning is used to generate and follow natural language instructions .
Approach:	They propose a pragmatic speaker that uses the base listener to simulate the interpretation of candidate descriptions and a listener that reasons counterfactually about alternative descriptions.
Outcome:	The proposed model improves state-of-the-art models for interpreting human instructions and speaker models in diverse settings.

Linguistic Bias in ChatGPT: Language Models Reinforce Dialect Discrimination (2024.emnlp-main)

Copied to clipboard

Eve Fleisig, Genevieve Smith, Madeline Bossi, Ishita Rustagi, Xavier Yin, Dan Klein

Enough Coin Flips Can Make LLMs Act Bayesian (2025.acl-long)

Challenge:	a large-scale study of linguistic bias exhibited by ChatGPT covers 10 dialects of English . standard varieties of English, especially SAE, dominate available training data .
Approach:	They use ChatGPT to generate models that default to "standard" varieties of English . they also use a feature annotation and native speaker evaluation to analyze the responses .
Outcome:	The proposed models default to "standard" varieties of English, but non-"standard" ones exhibit stereotyping, demeaning content, lack of comprehension, condescending responses.

Copied to clipboard

Ritwik Gupta, Rodolfo Corona, Jiaxin Ge, Eric Wang, Dan Klein, Trevor Darrell, David M. Chan

What’s Going On in Neural Constituency Parsers? An Analysis (N18-1)

Challenge:	Large language models exhibit the ability to generalize given few-shot examples in their input prompt, an emergent capability known as in-context learning.
Approach:	They investigate whether large language models use in-context learning to generalize given few-shot examples in their input prompt.
Outcome:	The proposed model can generalize given few-shot examples in their input prompt, an emergent capability known as in-context learning.

Copied to clipboard

David Gaddy, Mitchell Stern, Dan Klein

Challenge:	a number of differences have emerged between classical and modern constituency parsing approaches . structural components like grammars and feature-rich lexicons are becoming less central . recurrent neural networks have gained traction as a powerful and general purpose tool for representation .
Approach:	They propose a model that implicitly learns to encode much of the same information as grammars and lexicons in the past.
Outcome:	The proposed model outperforms state-of-the-art models under similar conditions.

Re3: Generating Longer Stories With Recursive Reprompting and Revision (2022.emnlp-main)

Copied to clipboard

Kevin Yang, Yuandong Tian, Nanyun Peng, Dan Klein

Cross-Domain Generalization of Neural Constituency Parsers (P19-1)

Challenge:	Recent work has generated short stories of several pages in length, but they are much shorter than typical short stories meant for human consumption.
Approach:	They propose a framework to generate long-range plot coherence and relevance by prompting a general-purpose language model and a language model.
Outcome:	The proposed framework generates stories of 2000-2500 words, compared to similar-length stories generated directly from the same model.

Copied to clipboard

Daniel Fried, Nikita Kitaev, Dan Klein

Voxel-informed Language Grounding (2022.acl-short)

Challenge:	Neural parsers perform well on in-domain benchmarks, but their performance degrades in well-understood ways.
Approach:	They analyze generalization on English and Chinese corpora to see if they can generalize to other domains.
Outcome:	The proposed neural parsers perform better on in-domain benchmarks than on out-of-domain corpora.

Copied to clipboard

Rodolfo Corona, Shizhan Zhu, Dan Klein, Trevor Darrell

Challenge:	Embodied robotic agents can be used to ground objects using 3D geometry . despite typically being paired with 2D images, natural language describes a fundamentally 3D world .
Approach:	They propose a model that leverages 3D geometric information to ground natural language . they show that VLG significantly improves grounding accuracy on SNARE .
Outcome:	The proposed model significantly improves grounding accuracy on SNARE, an object reference game task.

Constructing Taxonomies from Pretrained Language Models (2021.naacl-main)

Copied to clipboard

Catherine Chen, Kevin Lin, Dan Klein

LangProBe: a Language Program Benchmark (2025.findings-emnlp)

Challenge:	A variety of NLP tasks use taxonomic information, including question answering and information retrieval.
Approach:	They propose a method for constructing taxonomic trees using pretrained language models by incorporating web-retrieved glosses into the model.
Outcome:	The proposed model achieves 66.7 ancestor F1, a 20.0% relative increase over the previous best published model on English WordNet.

Copied to clipboard

Shangyin Tan, Lakshya A Agrawal, Arnav Singhvi, Liheng Lai, Michael J Ryan, Dan Klein, Omar Khattab, Koushik Sen, Matei Zaharia

A Deep Factorization of Style and Structure in Fonts (D19-1)

Challenge:	Composing language models into multi-step language programs is a mainstream paradigm for building AI systems, but tradeoffs in this space have only scarcely been studied before.
Approach:	They propose a benchmarking tool to evaluate the architectures and optimization strategies for language programs . they find that optimized language programs offer strong cost-quality Pareto improvement .
Outcome:	The proposed framework evaluates the impact of program architectures and optimizers on quality and cost.

Copied to clipboard

Nikita Srivatsan, Jonathan Barron, Dan Klein, Taylor Berg-Kirkpatrick

Challenge:	Using a variational inference procedure, we factor each training glyph into a combination of a character-specific content embedding and a latent font-specific style variable.
Approach:	They propose a deep factorization model that disentangles content from style by factorizing each training glyph into a latent content embedding and a learned embeddable character.
Outcome:	The proposed model outperforms a strong nearest neighbors baseline and state-of-the-art discriminative model on reconstructing missing glyphs from an unknown font given only a small number of observations.

Unsupervised Parsing via Constituency Tests (2020.emnlp-main)

Copied to clipboard

Steven Cao, Nikita Kitaev, Dan Klein

Understanding Game-Playing Agents with Natural Language Annotations (2022.acl-short)

Challenge:	Existing methods for unsupervised parsing rely on constituency tests . linguists can judge a sentence's grammatical validity by modifying it via some transformation .
Approach:	They propose a method for unsupervised parsing based on a constituency test . they specify a set of transformations and use an unsupervised neural acceptability model to make grammaticality decisions.
Outcome:	The proposed method achieves 62.8 F1 on the Penn Treebank test set, an improvement of 7.6 points over the previous best results.

Copied to clipboard

Nicholas Tomlin, Andre He, Dan Klein

Challenge:	a new dataset contains 10K human-annotated Go games.
Approach:	They propose to use natural language annotations to predict game terms from Go models . they use linear probing to predict domain-specific terms from the model's intermediate state representations .
Outcome:	The proposed model can predict domain-specific terms from human-annotated Go games . the proposed model is based on a dataset containing 10K human-rannotated games based in the game .

Pre-Learning Environment Representations for Data-Efficient Neural Instruction Following (P19-1)

Copied to clipboard

David Gaddy, Dan Klein

DOC: Improving Long Story Coherence With Detailed Outline Control (2023.acl-long)

Challenge:	Using logical forms, neural networks can sometimes require orders of magnitude more data to map from natural language instructions to state transitions (actions)
Approach:	They propose to map from natural language instructions to state transitions (actions) they augment a baseline learner with an initial environment-learning phase that uses observations of language-free state transition to induce a suitable latent representation of actions before processing the instruction-following training data.
Outcome:	The proposed model improves performance over systems whose representations are learned from limited instructional data alone.

Copied to clipboard

Kevin Yang, Dan Klein, Nanyun Peng, Yuandong Tian

Challenge:	Detailed Outline Control (DOC) framework improves long-range plot coherence . human evaluations of DOC show it outperforms strong Re3 on plot cohesion, outline relevance and interestingness .
Approach:	They propose a Detailed Outline Control framework to improve long-range plot coherence . the detailed outliner creates a more detailed, hierarchically structured outline . they propose doc with a detailed controller to ensure the more detailed outline is respected .
Outcome:	The proposed framework outperforms Re3 on plot coherence, outline relevance and interestingness.

Revisiting Entropy Rate Constancy in Text (2023.findings-emnlp)

Copied to clipboard

Vivek Verma, Nicholas Tomlin, Dan Klein