Papers by Tatsunori Hashimoto
Benchmarking Distributional Alignment of Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Language models are increasingly being used as simulacra for people, yet their ability to match the distribution of views of a specific demographic group remains uncertain. |
| Approach: | They construct a dataset expanding beyond political values and create human baselines for this task and evaluate the extent to which an LM can align with a particular group’s opinion distribution. |
| Outcome: | The proposed model can better describe opinion distributions than simulate demographic groups. |
When Do Pre-Training Biases Propagate to Downstream Tasks? A Case Study in Text Summarization (2023.eacl-main)
Copied to clipboard
Faisal Ladhak, Esin Durmus, Mirac Suzgun, Tianyi Zhang, Dan Jurafsky, Kathleen McKeown, Tatsunori Hashimoto
| Challenge: | Existing studies have shown that large language models contain linguistic and societal biases, but it is unclear how these biase amplify to downstream tasks. |
| Approach: | They investigate how name-nationality bias propagates from pre-training to downstream tasks . they show that these biases manifest themselves as hallucinations in summarization . |
| Outcome: | The proposed model can reduce the rate of hallucinations, but does not change the types of biases that do appear. |
Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | a recent study examines how epistemic markers of certainty, uncertainty, or evidentiality affect LMs' attitudes toward knowledge . accuracies of expressions of high certainty and low certainty are associated with questions . prior work focused on learning the mapping between internal probabilities of a model and an ordinal output . |
| Approach: | They develop a typology of epistemic markers and inject 50 markers into questions . they find that LMs are highly sensitive to epistemical markers in prompts . |
| Outcome: | The proposed model epistemology study shows that LMs are sensitive to epistemic markers in prompts . expressions of high certainty result in 7% decrease in accuracy, while factive verbs hurt performance . |
TempLM: Distilling Language Models into Template-Based Generators (2023.findings-acl)
Copied to clipboard
| Challenge: | Pretrained language models (PLMs) have greatly improved text generation, but they have also been known to produce unfaithful or inappropriate content. |
| Approach: | They propose a pretrained language model that is a template-based generator and uses it to generate a text. |
| Outcome: | The proposed model is more faithful than the original PLM and more fluent than prior template systems. |
Contrastive Decoding: Open-ended Text Generation as Optimization (2023.acl-long)
Copied to clipboard
Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, Mike Lewis
| Challenge: | Using a language model, maximum probability is a poor decoding objective because it produces short and repetitive text. |
| Approach: | They propose a reliable decoding approach that optimizes a contrastive objective subject to a plausibility constraint. |
| Outcome: | The proposed approach outperforms four strong decoding algorithms in automatic and human evaluations across wikipedia, news and story domains. |
Spurious Correlations in Reference-Free Evaluation of Text Generation (2022.acl-long)
Copied to clipboard
| Challenge: | Recent work suggests that reference-free evaluation metrics may rely on spurious correlations with human judgments. |
| Approach: | They propose to use model-based, reference-free evaluation metrics to evaluate natural language generation systems. |
| Outcome: | The proposed metrics achieve high correlations with human judgments, but they may not be robust enough to evaluate their efficacy and robustness. |
s1: Simple test-time scaling (2025.emnlp-main)
Copied to clipboard
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candes, Tatsunori Hashimoto
| Challenge: | OpenAI’s o1 model showed this capability but did not publicly share its methodology, leading to many replication efforts. |
| Approach: | They curate a small dataset s1K with 1,000 reasoning questions based on three criteria we validate through ablations: difficulty, diversity, and quality. |
| Outcome: | The proposed model exceeds o1-preview on competition math questions by up to 27% (MATH and AIME24). |
Contrastive Error Attribution for Finetuned Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods for error tracing do not detect faithfulness errors in NLG datasets. |
| Approach: | They propose a framework to identify and remove low-quality training instances that lead to undesirable outputs. |
| Outcome: | The proposed method outperforms existing methods for detecting faithfulness errors in NLG datasets. |
Measuring Conversational Uptake: A Case Study on Student-Teacher Interactions (2021.acl-long)
Copied to clipboard
Dorottya Demszky, Jing Liu, Zid Mancenido, Julie Cohen, Heather Hill, Dan Jurafsky, Tatsunori Hashimoto
| Challenge: | Despite extensive research showing the positive impact of uptake on student learning and achievement, there is little evidence that it is effective in teaching. |
| Approach: | They propose a framework for computationally measuring uptake by releasing a dataset of student-teacher exchanges extracted from US math classroom transcripts annotated for uptake . they formalize uptake as pointwise Jensen-Shannon Divergence (pJSD) and conduct a linguistically-motivated comparison of different unsupervised measures. |
| Outcome: | The proposed framework outperforms baseline measures in identifying uptake phenomena like question answering and reformulation. |
Removing RLHF Protections in GPT-4 via Fine-Tuning (2024.naacl-short)
Copied to clipboard
| Challenge: | Large language models (LLMs) have increased in their capabilities, which increases their potential for dual use. |
| Approach: | They show that fine-tuning can remove RLHFprotections with as few as 340 examples and a 95% success rate. |
| Outcome: | The proposed method removes RLHFprotections with as few as 340 examples and a 95% success rate on non-censored outputs. |
Privacy-Preserving Domain Adaptation of Semantic Parsers (2023.acl-long)
Copied to clipboard
| Challenge: | Task-oriented dialogue systems often assist users with personal or confidential matters . a lack of privacy controls prevents developers from observing actual usage . authors propose a method to generate realistic user utterances synthetically without compromising privacy . |
| Approach: | They propose a method which generates latent semantic parses and generates utterances based on the parses. |
| Outcome: | The proposed method improves MAUVE by 2.5X and parse tree function-type overlap by 1.3X . it also shows gains of 8.5% points on its accuracy with the new feature . |