Papers by Saku Sugawara
How Well Do Multi-hop Reading Comprehension Models Understand Date Information? (2022.aacl-short)
Copied to clipboard
| Challenge: | Existing multi-hop reading comprehension datasets have reasoning shortcuts that can be used to answer comparison questions without performing multi- hop reasoning. |
| Approach: | They propose a dataset with three probing tasks in addition to the main question . they then evaluate the model's ability to understand date information . |
| Outcome: | The proposed model performs well in date comparison and number subtraction tasks. |
Label Effects: Shared Heuristic Reliance in Trust Assessment by Humans and LLM-as-a-Judge (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly used as automated evaluators . et al., 2024: strong labels can foster trust but also undermine it . |
| Approach: | They show that LLMs' source labels bias trust judgments by humans . they use eye-tracking data to analyze LLM internal states during judgment . |
| Outcome: | The proposed model is biased by disclosed source labels, the authors show . eye-tracking data show humans rely heavily on source labels for judgments . |
On Degrees of Freedom in Defining and Testing Natural Language Understanding (2023.findings-acl)
Copied to clipboard
| Challenge: | Large-scale pretrained language models are performing increasingly well at various tasks and offering real-world applications. |
| Approach: | They propose to define NLU as an inductive evidence that the test subject understands the language sufficiently well to meet stakeholder objectives. |
| Outcome: | The proposed framework can be used to design credible tests and facilitate scientific communication. |
Cross-Modal Similarity-Based Curriculum Learning for Image Captioning (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing image captioning approaches treat image-caption pairs indistinctly without considering the differences in their learning difficulties. |
| Approach: | They propose a pretrained vision–language model that measures cross-modal similarity and a model that uses cross-module similarity to measure the difficulty of captioning. |
| Outcome: | The proposed model achieves superior performance and competitive convergence speed to baselines without incurring additional training costs. |
Analyzing the Effectiveness of the Underlying Reasoning Tasks in Multi-hop Question Answering (2023.findings-eacl)
Copied to clipboard
| Challenge: | Existing studies have utilized underlying reasoning (UR) tasks in multi-hop question answering datasets to explain the predicted answers and evaluate models' reasoning abilities. |
| Approach: | They analyze UR tasks in QA datasets to determine their effectiveness . they find that UR task is helpful in preventing reasoning shortcuts . |
| Outcome: | The proposed model improves QA performance, reasoning shortcuts, and robustness on adversarial questions. |
A Dual-Task Paradigm to Investigate Sentence Comprehension Strategies in Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods do not directly target the balance between memory and sentence processing, which is central to human working memory. |
| Approach: | They propose a dual-task paradigm that combines arithmetic computation with sentence comprehension . they show a greater accuracy gap between plausible sentences and implausible sentences . |
| Outcome: | The proposed paradigm shows that plausibility-based comprehension mirrors humans’ rational inference. |
Evaluating the Rationale Understanding of Critical Reasoning in Logical Reading Comprehension (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent large language models (e.g., InstructGPT) struggle to answer subquestions written for the incorrect options of the main questions, implying that they have a limited capability for explaining why incorrect alternatives should be eliminated. |
| Approach: | They crowdsource free-form rationale for main questions and generate subquestion texts that explain why we should select or eliminate answer options. |
| Outcome: | The proposed dataset shows that large language models struggle to answer the subquestions even if they are able to answer main questions correctly. |
What Makes Reading Comprehension Questions Difficult? (2022.acl-long)
Copied to clipboard
| Challenge: | a recent study shows that natural language understanding benchmarks are not able to measure future progress . a crowdsourcing approach is needed to collect diverse examples without sacrificing diversity or coverage. |
| Approach: | They crowdsource multiple-choice reading comprehension questions for passages from seven sources . they find passage source, length, and readability measures do not significantly affect question difficulty . |
| Outcome: | The results show that passage source, length, and readability measures do not significantly affect question difficulty. |
TactfulToM: Do LLMs have the Theory of Mind ability to understand White Lies? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies explore Large Language Models’ (LLMs) performance on Theory of Mind (ToM) reasoning tasks, but research on ToM abilities that require more nuanced social context is limited, such as white lies. |
| Approach: | They propose a novel English benchmark to evaluate Large Language Models’ ability to understand white lies within real-life conversations and reason about prosocial motivations behind them. |
| Outcome: | The proposed model outperforms state-of-the-art models on ToM reasoning tasks and reveals significant gaps between humans and LLMs. |
CxMP: A Linguistic Minimal-Pair Benchmark for Evaluating Constructional Understanding in Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Understanding language acquisition in language models remains an open question, yet many benchmarks focus on grammatical acceptability, with far less attention to interpreting meanings conveyed by grammatological forms. |
| Approach: | They propose a benchmark to evaluate constructional understanding in language models using a controlled minimal-pair. |
| Outcome: | The proposed benchmarks show that understanding of constructions develops more slowly and remains limited even in large language models (LLMs). |
Debiasing Masks: A New Framework for Shortcut Mitigation in NLU (2022.emnlp-main)
Copied to clipboard
| Challenge: | Debiasing language models from unwanted behaviors in natural language understanding datasets is a topic with increasing interest in the NLP community. |
| Approach: | They propose a method to debiase language models from unwanted behaviors in NLU tasks by identifying pruning masks that can be applied to a finetuned model. |
| Outcome: | The proposed method shows superior performance and performance over standard methods. |
Are Checklists Really Useful for Automatic Evaluation of Generative Tasks? (2025.emnlp-main)
Copied to clipboard
| Challenge: | evaluators using large language models face ambiguous criteria and inconsistent evaluations. |
| Approach: | They investigate whether checklists should be used for all questions or selectively . they generate checklists using six methods and evaluate their effectiveness across eight models . |
| Outcome: | The proposed method improves evaluation performance in pairwise comparisons while ignoring human-written criteria. |
What Ingredients Make for an Effective Crowdsourcing Protocol for Difficult NLU Data Collection Tasks? (2021.acl-long)
Copied to clipboard
| Challenge: | Despite the importance of datasets for natural language understanding, there has been little attention on crowdsourcing methods for collecting datasets. |
| Approach: | They compare the effectiveness of crowdsourcing methods for boosting NLU example difficulty with training crowdworkers instead of expert judgments. |
| Outcome: | The proposed method is ineffective for boosting NLU example difficulty, but it is not effective for training crowdworkers and qualifying workers based on expert judgments. |
C2: Scalable Rubric-Augmented Reward Modeling from Binary Preferences (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for rubric-augmented verification require costly rubric annotations, limiting scalability. |
| Approach: | They propose a framework that allows a reward model to collaborate critically with a rubric generator trained solely from binary preferences. |
| Outcome: | The proposed framework outperforms reasoning reward models trained on binary preferences with gains of up to 6.5 points on RM-Bench and 6.0 points length-controlled win rate on AlpacaEval 2.0. |
Modeling Overregularization in Children with Small Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing research has analyzed regularization in language acquisition only by modeling word inflection directly, which is unnatural in light of human language acquisition. |
| Approach: | They hypothesize that language models that imitate errors children make during language acquisition have a learning process more similar to humans. |
| Outcome: | The proposed model shows child-like U-shaped learning curves clearly for certain verbs, but the preferences for types of overgeneralization did not fully match the observations in children. |
Improving the Robustness of QA Models to Challenge Sets with Variational Question-Answer Pair Generation (2021.acl-srw)
Copied to clipboard
| Challenge: | Existing data augmentation methods for reading comprehension lack robustness to challenge sets whose distribution is different from that of training sets. |
| Approach: | They propose a question-answer pair generation method that generates multiple diverse QA pairs from a paragraph to mitigate this problem. |
| Outcome: | The proposed model improves the accuracy of 12 challenge sets and the in-distribution accuracy. |
Embracing Ambiguity: Shifting the Training Target of NLI Models (2021.acl-short)
Copied to clipboard
| Challenge: | Previously, it was common to disregard ambiguity as noise or as a sign of poor quality data. |
| Approach: | They propose to train on the estimated label distribution of annotators in a NLI task . they use a learning loss based on this ambiguity distribution instead of gold-labels . |
| Outcome: | The proposed training method reduces divergence scores on a trial dataset . the study shows that targeting the ambiguity distribution can improve performance . |
Rationale-Aware Answer Verification by Pairwise Self-Evaluation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Current approaches to train verifier models neglecting flawed rationales, resulting in an unreliable verifier. |
| Approach: | They propose a method for selecting valid rationales from candidates by iteratively applying pairwise self-evaluation using the same LLM that generates the solutions. |
| Outcome: | The proposed method outperforms training methods on three reasoning benchmarks. |
Development of Numerical Error Detection Tasks to Analyze the Numerical Capabilities of Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Existing language models are difficult to detect numerical errors because of their finite set of tokens. |
| Approach: | They use a benchmark dataset to classify numerical errors using automatically generated numerical errors and investigate their ability to detect errors. |
| Outcome: | The proposed model performs well in the numerical error detection task, but not as accurate as humans. |
Benchmarking Machine Reading Comprehension: A Psychological Perspective (2021.eacl-main)
Copied to clipboard
| Challenge: | MRC is a task that tests the ability of a machine to read and understand unstructured text. |
| Approach: | They propose a theoretical basis for the design of MRC datasets based on psychology and psychometrics and propose shortcut-proof questions and explanations as a part of the task design. |
| Outcome: | The proposed datasets should evaluate the model's ability to understand context-dependent situations and ensure substantive validity by shortcut-proof questions and explanation as a part of the task design. |
What Makes Language Models Good-enough? (2024.findings-acl)
Copied to clipboard
| Challenge: | Psycholinguistic research suggests that humans may build a representation of linguistic input that is ‘good-enough’ for the task at hand. |
| Approach: | They create a good-enough language processing evaluation dataset to test the effects of two plausibility types, eight construction types, and three degrees of memory cost on language processing. |
| Outcome: | The proposed model evaluation against the annotated GELP reveals that models with shallower depth and fewer heads exhibit good-enough performance. |
Possible Stories: Evaluating Situated Commonsense Reasoning under Multiple Possible Scenarios (2022.coling-1)
Copied to clipboard
| Challenge: | Current studies in natural language processing do not focus on situated commonsense reasoning under multiple possible scenarios. |
| Approach: | They frame a scenario task by asking multiple questions with the same set of possible endings as candidate answers, given a short story text. |
| Outcome: | The proposed dataset shows that even strong pretrained models struggle to answer the questions consistently, highlighting that the highest accuracy in an unsupervised setting (60.2%) is far behind human accuracy (92.5%). |
Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps (2020.coling-main)
Copied to clipboard
| Challenge: | Existing multi-hop question answering datasets do not provide a complete explanation for the reasoning process from the question to the answer. |
| Approach: | They propose a multi-hop question answering dataset that uses structured and unstructured data to test reasoning skills. |
| Outcome: | The proposed dataset ensures multi-hop reasoning while being challenging for multi-models. |
What Makes Reading Comprehension Questions Easier? (D18-1)
Copied to clipboard
| Challenge: | Recent studies have shown that questions require a deeper understanding of language to answer beyond using superficial cues. |
| Approach: | They propose to use simple heuristics to split MRC datasets into easy and hard subsets and manually annotate questions from each subset with validity and reasoning skills to investigate which skills explain the difference between easy and harder questions. |
| Outcome: | The proposed model performs better for hard and easy questions than for easy questions. |
Can Language Models Induce Grammatical Knowledge from Indirect Evidence? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in language models have shown remarkable progress in various tasks. |
| Approach: | They introduce a dataset that incorporates wug words and inject them into pretraining data and evaluate them on evaluation data. |
| Outcome: | The proposed model does not induce grammatical knowledge even after repeated exposure to instances with the same structure but differing only in lexical items from evaluation instances in certain language phenomena. |
Probing Physical Reasoning with Counter-Commonsense Context (2023.acl-short)
Copied to clipboard
| Challenge: | Recent language models have succeeded in tasks that do not require contextual reasoning, such as size comparison and prediction of event frequency. |
| Approach: | They propose to use a dataset to measure physical commonsense to infer size relationships between objects in contexts that fit physical common sense. |
| Outcome: | The proposed dataset consists of contexts that fit physical commonsense and those that don't . results show that language models fail to use verbs and make incorrect judgments . |