Papers by Keisuke Sakaguchi
proScript: Partially Ordered Scripts Generation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Scripts represent structured commonsense knowledge about prototypical events in everyday situations/scenarios such as bake a cake. |
| Approach: | They collect 6.4k crowdsourced partially ordered scripts and develop models that combine language generation and graph structure prediction to generate scripts. |
| Outcome: | The proposed models perform well on two tasks: edge prediction and script generation. |
A Call for Clarity in Beam Search: How It Works and When It Stops (2024.lrec-main)
Copied to clipboard
| Challenge: | Empirical results show that a modified beam decoding implementation improves decoding performance of strong, neural language generation models. |
| Approach: | They propose a modification to a beam decoding implementation that generalizes the stopping criterion and provides flexibility to the depth of search. |
| Outcome: | The proposed method improves decoding performance of strong models on news text summarization and machine translation over diverse language pairs with negligible inference slowdown. |
Twist Decoding: Diverse Generators Guide Each Other (2022.emnlp-main)
Copied to clipboard
Jungo Kasai, Keisuke Sakaguchi, Ronan Le Bras, Hao Peng, Ximing Lu, Dragomir Radev, Yejin Choi, Noah A. Smith
| Challenge: | Using a variety of language generation models, ensembling models is challenging during inference. |
| Approach: | They propose a method that decodes text models that do not assume a shared vocabulary, tokenization or generation order. |
| Outcome: | The proposed method outperforms models decoded in isolation over various scenarios. |
A Dataset for Tracking Entities in Open Domain Procedural Text (2020.emnlp-main)
Copied to clipboard
Niket Tandon, Keisuke Sakaguchi, Bhavana Dalvi, Dheeraj Rajagopal, Peter Clark, Michal Guerquin, Kyle Richardson, Eduard Hovy
| Challenge: | Existing tasks require only a small set of attributes to track state changes in procedural text. |
| Approach: | They propose a task where given a procedural text as input, the task is to generate a set of state change tuples for each step. |
| Outcome: | The proposed task generates state change tuples from a set of pre-defined attributes for each step and predicts them from an open vocabulary. |
LLMs Faithfully and Iteratively Compute Answers During CoT: A Systematic Analysis With Multi-step Arithmetics (2026.findings-eacl)
Copied to clipboard
Keito Kudo, Yoichi Aoki, Tatsuki Kuribayashi, Shusaku Sone, Masaya Taniguchi, Ana Brassard, Keisuke Sakaguchi, Kentaro Inui
| Challenge: | Specifically, we examine when the LLMs’ answer is (pre)determined, especially before the CoT begins or after, and how strongly the information from CoT specifically has a causal effect on the final answer. |
| Approach: | They examine when the LLMs’ answer is (pre)determined, especially before the CoT begins or after, and how strongly the information from CoT specifically has a causal effect on the final answer. |
| Outcome: | The proposed model can generate reasoning chains while generating the reasoning chain on the fly. |
Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand (2022.naacl-main)
Copied to clipboard
Jungo Kasai, Keisuke Sakaguchi, Ronan Le Bras, Lavinia Dunagan, Jacob Morrison, Alexander Fabbri, Yejin Choi, Noah A. Smith
| Challenge: | Recent advances on models and metrics should benefit and inform each other, authors argue . bidimensional leaderboards allow for fast, accurate evaluation of language generation models . |
| Approach: | They propose a bidimensional leaderboard that tracks progress in language generation models and metrics for their evaluation. |
| Outcome: | The proposed leaderboards track progress in language generation models and metrics for their evaluation. |
Self-Training Meets Consistency: Improving LLMs’ Reasoning with Consistency-Driven Rationale Evaluation (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches labeled rationales that produce correct answers as appropriate for training but one measure risks misjudging rationale quality, leading models to learn flawed reasoning patterns. |
| Approach: | They propose a framework that evaluates rationales through follow-up questions and leverages this evaluation to guide its training. |
| Outcome: | The proposed framework improves robustness and correctness of rationales and reasoning abilities compared to previous self-training approaches. |
Weight-based Analysis of Detokenization in Language Models: Understanding the First Stage of Inference Without Inference (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing evidence for the stages-of-inference hypothesis is that early layers of language models map their subword tokenized input to more meaningful representations that form the model’s “inner vocabulary”. |
| Approach: | They propose an analytical decomposition of first-layer attention in language models that quantifies the relative contributions of position-related, token-related and mixed effects. |
| Outcome: | The proposed analysis yields interpretable terms that quantify the relative contributions of position-related, token-related and mixed effects. |
Empirical Investigation of Neural Symbolic Reasoning Strategies (2023.findings-eacl)
Copied to clipboard
Yoichi Aoki, Keito Kudo, Tatsuki Kuribayashi, Ana Brassard, Masashi Yoshikawa, Keisuke Sakaguchi, Kentaro Inui
| Challenge: | Neural reasoning accuracy improves when generating intermediate reasoning steps. |
| Approach: | They decompose the reasoning strategy w.r.t. step granularity and chaining strategy. |
| Outcome: | The proposed reasoning strategy significantly affects performance in a symbolic reasoning dataset. |
Quantifying the Influence of Evaluation Aspects on Long-Form Response Assessment (2025.coling-main)
Copied to clipboard
| Challenge: | Evaluating the outputs of large language models (LLMs) on long-form generative tasks remains challenging. |
| Approach: | They propose to compute an overall quality score as a weighted average of factuality, informative-ness, and formality as compared to previous metrics. |
| Outcome: | The proposed method achieves stronger correlations with human judgments compared to previous metrics. |
ELQA: A Corpus of Metalinguistic Questions and Answers about English (2023.acl-long)
Copied to clipboard
| Challenge: | ELQA corpus is metalinguistic—it consists of language about language. |
| Approach: | They present a corpus of questions and answers in and about the English language . they use a free-form question answering task and multiple LLMs to analyze their capacity . |
| Outcome: | The ELQA corpus covers grammar, meaning, fluency, and etymology . the results can be used to investigate metalinguistic capabilities of NLU models . |
I2D2: Inductive Knowledge Distillation with NeuroLogic and Self-Imitation (2023.acl-long)
Copied to clipboard
Chandra Bhagavatula, Jena D. Hwang, Doug Downey, Ronan Le Bras, Ximing Lu, Lianhui Qin, Keisuke Sakaguchi, Swabha Swayamdipta, Peter West, Yejin Choi
| Challenge: | Empirical results suggest that scale is not the only way to build commonsense capabilities. |
| Approach: | They propose a commonsense distillation framework that can achieve a competitive level of commonsensing without relying on the benefits of scale. |
| Outcome: | The proposed framework breaks the dependence on the extreme-scale teacher model with two innovations: (1) the novel adaptation of NeuroLogic Decoding to enhance the generation quality of the weak, off-the-shelf language models, and (2) self-imitation learning to iteratively learn from the model’s own enhanced commonsense acquisition capabilities. |
Transparent Human Evaluation for Image Captioning (2022.naacl-main)
Copied to clipboard
Jungo Kasai, Keisuke Sakaguchi, Lavinia Dunagan, Jacob Morrison, Ronan Le Bras, Yejin Choi, Noah A. Smith
| Challenge: | Recent work has demonstrated that image captioning is a complex task that requires a large amount of human input. |
| Approach: | They develop a human evaluation protocol for image captioning models based on machine- and human-generated captions on the MSCOCO dataset. |
| Outcome: | The proposed model improves CLIPScore, a recent metric that uses image features, and improves human judgments because it is more sensitive to recall. |
Rubrik’s Cube: Testing a New Rubric for Evaluating Explanations on the CUBE dataset (2025.acl-long)
Copied to clipboard
Diana Galvan-Sosa, Gabrielle Gaudeau, Pride Kavumba, Yunmeng Li, Hongyi Gu, Zheng Yuan, Keisuke Sakaguchi, Paula Buttery
| Challenge: | Large-Language Models (LLMs) are increasingly being used in explanation generation tasks due to their unreliability. |
| Approach: | They propose a rubric and a dataset of 26k explanations written and quality-annotated using the rubric by humans and six open- and closed-source LLMs to test their proposed rubric. |
| Outcome: | The proposed rubric and CUBE dataset focuses on reasoning and language tasks and provides the necessary diversity to test it. |
Do Deep Neural Networks Capture Compositionality in Arithmetic Reasoning? (2023.eacl-main)
Copied to clipboard
Keito Kudo, Yoichi Aoki, Tatsuki Kuribayashi, Ana Brassard, Masashi Yoshikawa, Keisuke Sakaguchi, Kentaro Inui
| Challenge: | Using a pre-trained dataset, we examine how well recent neural models capture compositionality in symbolic reasoning tasks. |
| Approach: | They propose a skill tree on compositionality that defines hierarchical levels of complexity along with three compositionality dimensions: systematicity, productivity, and substitutivity. |
| Outcome: | The proposed model struggled most with systematicity, performing poorly even with relatively simple compositions. |
How Well Do Vision Models Encode Diagram Attributes? (2024.acl-srw)
Copied to clipboard
Haruto Yoshida, Keito Kudo, Yoichi Aoki, Ryota Tanaka, Itsumi Saito, Keisuke Sakaguchi, Kentaro Inui
| Challenge: | Experimental results show vision models struggle to identify diagram attributes such as node colors and shapes, along with edge colors and connection patterns. |
| Approach: | They evaluated vision models and retrieving diagrams using text queries to determine how well they recognize diagram attributes and edge connection patterns. |
| Outcome: | The models can recognize node colors, shapes, and edge colors, but struggle to identify differences in edge connection patterns that play a pivotal role in the semantics of diagrams. |
Test-time Augmentation for Factual Probing (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to improve factual probing are relation-specific and do not generalize to unseen relation types. |
| Approach: | They propose to use test-time augmentation to augment and ensemble prompts at test time to reduce sensitivity to prompt variations. |
| Outcome: | The proposed method improves model confidence, but for other models, it leads to degradation. |
Uncertain Natural Language Inference (2020.acl-main)
Copied to clipboard
| Challenge: | Existing categorical bin assignments for NLI are not effective for entailment inference. |
| Approach: | They propose a refinement of Natural Language Inference that targets direct prediction of subjective probability assessments. |
| Outcome: | The proposed model can be used in pre-training and correlates well with humans. |
WIQA: A dataset for “What if...” reasoning over procedural text (D19-1)
Copied to clipboard
| Challenge: | a dataset of “What if...” questions is available for procedural text comprehension . we present the dataset as an open challenge to the community . |
| Approach: | They propose a dataset of “What if...” questions over procedural text . they use paragraphs annotated with multiple influence graphs to create the questions . |
| Outcome: | The proposed dataset achieves 73.8% accuracy, well below the human performance of 96.3%. |
Efficient Online Scalar Annotation with Bounded Support (P18-1)
Copied to clipboard
| Challenge: | Existing methods for efficiently eliciting scalar annotations for dataset construction and system quality estimation by human judgments are not shown. |
| Approach: | They propose a method for efficiently eliciting scalar annotations by human judgments. |
| Outcome: | The proposed method leads to increased correlation with ground truth, suggesting it is an improved mechanism for dataset creation and manual system evaluation. |
First Heuristic Then Rational: Dynamic Use of Heuristics in Language Model Reasoning (2024.emnlp-main)
Copied to clipboard
Yoichi Aoki, Keito Kudo, Tatsuki Kuribayashi, Shusaku Sone, Masaya Taniguchi, Keisuke Sakaguchi, Kentaro Inui
| Challenge: | Explicit multi-step reasoning is widely adopted to improve the performance of language models. |
| Approach: | They propose a systematic reasoning strategy that LMs use to solve multi-step reasoning tasks. |
| Outcome: | The proposed strategy improves the performance of language models by combining heuristics with rational strategies. |
The Universal Decompositional Semantics Dataset and Decomp Toolkit (2020.lrec-1)
Copied to clipboard
Aaron Steven White, Elias Stengel-Eskin, Siddharth Vashishtha, Venkata Subrahmanyan Govindarajan, Dee Ann Reisinger, Tim Vieira, Keisuke Sakaguchi, Sheng Zhang, Francis Ferraro, Rachel Rudinger, Kyle Rawlins, Benjamin Van Durme
| Challenge: | Decompositional semantics is a method of crowd-sourcing semantic annotations while retaining high interannotator agreement. |
| Approach: | They present the Universal Decompositional Semantics dataset (v1.0) they propose a decomposition-aligned approach to semantic annotation that uses simple questions to answer . |
| Outcome: | The dataset is bundled with the Decomp toolkit (v0.1) both datasets are publicly available at http://decomp.io. |
Language Models can Categorize System Inputs for Performance Analysis (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing evaluations only provide a single numerical score for broad categories. |
| Approach: | They propose to automatically search for finer-grained categories based on inputs where a system performs well or poorly and describe them in natural language. |
| Outcome: | The proposed model compares LLaMA 3-70B and Claude 3 Opus with similar Elo ratings on Chatbot Arena. |