Papers by Yusuke Sakai

Tomoyuki Jinno, Kazuki Hayashi, Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe

Challenge:	Existing studies show that MBR decoding improves model generation performance . however, the theoretical underpinnings of these results remain uncertain .
Approach:	They propose a theoretical interpretation of MBR decoding from the perspective of bias–diversity decomposition.
Outcome:	The proposed method improves the quality estimation of hypotheses by decomposing bias and diversity into two main factors.

Copied to clipboard

Multilingual Dialogue Generation and Localization with Dialogue Act Scripting (2025.emnlp-main)

Challenge:	Existing knowledge probes for pre-trained language models exhibit quadratic time complexity, limiting the size of knowledge graphs used for probing.
Approach:	They propose an embedding-based relational probe that evaluates pre-trained language models' factual knowledge retrieval capabilities.
Outcome:	The proposed probe achieves effective time complexity of linear order O(n), supports rank-based evaluation metrics including Hit@k, handles multi-token entity names and enables probing whilst disambiguating homographic tail-entity names.

Copied to clipboard

Justin Vasselli, Eunike Andriani Kardinata, Yusuke Sakai, Taro Watanabe

“Yuki Gets Sushi, David Gets Steak?”: Uncovering Gender and Racial Biases in LLM-Based Meal Recommendations (2026.eacl-long)

Challenge:	Existing approaches to training or evaluating non-English dialogue datasets often introduce artifacts that reduce their naturalness and cultural appropriateness.
Approach:	They propose a structured framework for encoding, localizing, and generating multilingual dialogues from abstract intent representations.
Outcome:	The proposed framework outperforms translation models in Italian, German, and Chinese on cultural relevance, coherence, and situational appropriateness.

Copied to clipboard

Xuefeng Wei, Xuan Zhou, Yusuke Sakai, Taro Watanabe

Centroid-Based Efficient Minimum Bayes Risk Decoding (2024.findings-acl)

Challenge:	Large language models inherit and amplify societal biases related to gender and race.
Approach:	They use a USChainMains dataset to evaluate group bias in Large Language Models . they found that LLMs recommend meals with higher levels of adverse nutrients for names associated with Black, Hispanic, or male individuals .
Outcome:	The proposed model scales improves overall recommendation healthfulness but is insufficient to eliminate the healthfulness gap between demographic groups.

Copied to clipboard

Hiroyuki Deguchi, Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe, Hideki Tanaka, Masao Utiyama

Tonguescape: Exploring Language Models Understanding of Vowel Articulation (2025.naacl-long)

Challenge:	Minimum Bayes risk (MBR) decoding requires quadratic time since it computes the expected score between a translation hypothesis and all reference translations.
Approach:	They propose a centroid-based MBR decoding method that clusters the translations in the feature space and calculates the expected score using the centroids of each cluster.
Outcome:	The proposed method outperforms vanilla MBR decoding in translation quality by up to 0.5 COMET in the WMT’22 EnJa, EnDe, EnZh, and WMT'23 Enja translation tasks.

Copied to clipboard

Haruki Sakajo, Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe

One Single Hub Text Breaks CLIP: Identifying Vulnerabilities in Cross-Modal Encoders via Hubness (2026.acl-long)

Challenge:	a study shows that language models can explain vowel pronunciation based on tongue positions . a visual LM can explain the relationship between vowels and tongue positions, but it is unclear whether they align textual information with visual information.
Approach:	They created video and image datasets from MRI data to examine if LMs associate real tongue positions with vowel articulation.
Outcome:	The proposed model can explain vowel pronunciation and the correlation between vowels and tongue positions as textual knowledge.

Copied to clipboard

Hiroyuki Deguchi, Katsuki Chousa, Yusuke Sakai

How to Make the Most of LLMs’ Grammatical Knowledge for Acceptability Judgments (2025.naacl-long)

Challenge:	et al., 2010) show that hub embeddings are close to many unrelated examples in high-dimensional embeddable spaces . cross-modal encoders that project different modalities into a shared space are useful for cross-module applications .
Approach:	They propose a method for identifying the hub embedding and its corresponding hub text . they use images to evaluate cross-modal encoders that project different modalities into a shared space .
Outcome:	The proposed method can identify a single hub embedding and its corresponding hub text . it achieves comparable or higher similarity scores than human-written reference captions in many images .

Copied to clipboard

Yusuke Ide, Yuto Nishida, Justin Vasselli, Miyu Oba, Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe

Rethinking Evaluation Metrics for Grammatical Error Correction: Why Use a Different Evaluation Process than Human? (2025.acl-short)

Challenge:	Conventional approaches compare sentence probabilities directly, but large language models (LLMs) provide nuanced evaluation methods using prompts and templates.
Approach:	They propose to derive acceptability judgments from large language models using prompts and templates to comprehensively evaluate their grammatical knowledge.
Outcome:	The proposed methods excel in different linguistic phenomena, suggesting they access different aspects of the LLMs’ grammatical knowledge.

Copied to clipboard

Takumi Goto, Yusuke Sakai, Taro Watanabe

Syntactic Text Processing

Challenge:	Existing automatic evaluation metrics are based on procedures that diverge from human evaluation.
Approach:	They propose to aggregate automatic evaluation metrics to bridge this gap . they propose to use edit-based metrics, -gram based metrics and sentence-level metrics to find the best ranking system.
Outcome:	The proposed method outperforms existing metrics on the SEEDA benchmark and improves edit-based metrics, -gram based metrics and sentence-level metrics.

HalluCitation Matters: Revealing the Impact of Hallucinated References with 300 Hallucinated Papers in ACL Conferences (2026.acl-long)

Copied to clipboard

Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe

Reasoning

Challenge:	citations that do not correspond to any existing work are a serious concern to scientific reliability and credibility.
Approach:	They analyze papers published at ACL, NAACL, and EMNLP in 2024 and 2025 . they identify 300 papers with at least one HalluCitation, most of which were published in 2025.
Outcome:	The authors analyze papers published at ACL, NAACL, and EMNLP in 2024 and 2025 . they find that nearly 300 papers contain at least one HalluCitation, most of which were published in 2025.

AdTEC: A Unified Benchmark for Evaluating Text Quality in Search Engine Advertising (2025.naacl-long)

Copied to clipboard

Peinan Zhang, Yusuke Sakai, Masato Mita, Hiroki Ouchi, Taro Watanabe

Towards Singable Lyrics Translation Using Large Language Models (2026.eacl-srw)

Challenge:	Existing pre-trained language models outperform them in certain domains, indicating that there is significant potential for further improvement in this area.
Approach:	They propose to use pre-trained language models to evaluate ad texts from multiple perspectives within real-world advertising operations to define five tasks and construct a Japanese dataset.
Outcome:	The proposed benchmark outperforms existing pre-trained language models in several tasks, but humans outperformed them in certain domains.

Copied to clipboard

Liu Hanze, Yusuke Sakai, Taro Watanabe

Diagnosing Vision Language Models’ Perception by Leveraging Human Methods for Color Vision Deficiencies (2026.eacl-long)

Challenge:	Existing studies on lyrics translation have relied on fine-tuning open-source language models.
Approach:	They examine a multilingual lyrics translation dataset and apply prompting methods to large language models to evaluate singability.
Outcome:	The proposed methods improve singability and naturalness, compared to naive translation, the authors show . human evaluations using songs created from translated lyrics show that complex prompting strategies improve singable naturalness .

Copied to clipboard

Kazuki Hayashi, Shintaro Ozaki, Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe

Multimodality

Challenge:	Large-scale Vision-Language Models (LVLMs) are being deployed in real-world settings that require visual inference.
Approach:	They evaluate LVLMs' ability to account for variation in color perception using the Ishihara Test.
Outcome:	The proposed models fail to reproduce the perceptual outcomes experienced by affected individuals and default to normative color perception.

Assessing the Effect of Context in Multi-domain Acceptability Judgment (2026.findings-acl)

Copied to clipboard

Eunike Andriani Kardinata, Yusuke Sakai, Taro Watanabe

Challenge:	Existing studies evaluate sentences in isolation and do not consider how context influences LLM acceptability judgments.
Approach:	They examine how contextual cues affect model-generated acceptability ratings across multiple domains and several LLMs, using different forms of domain-specific contextual cueeds to situate sentences in intended usage settings.
Outcome:	The findings support the development of more context-aware evaluation frameworks.

Dictionaries to the Rescue: Cross-Lingual Vocabulary Transfer for Low-Resource Languages Using Bilingual Dictionaries (2025.findings-acl)

Copied to clipboard

Haruki Sakajo, Yusuke Ide, Justin Vasselli, Yusuke Sakai, Yingtao Tian, Hidetaka Kamigaito, Taro Watanabe

Simul-MuST-C: Simultaneous Multilingual Speech Translation Corpus Using Large Language Model (2024.emnlp-main)

Challenge:	Existing approaches to cross-lingual vocabulary transfer face challenges when dealing with low-resource languages.
Approach:	They propose a dictionary-based crosslingual vocabulary transfer method that leverages bilingual dictionaries, which are available for many languages thanks to descriptive linguists.
Outcome:	The proposed method outperforms existing methods for low-resource languages.

Copied to clipboard

Mana Makinae, Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe

Multilinguality

Challenge:	Simultaneous speech translation (SiST) begins translating before the entire source input is received.
Approach:	They propose a dataset that rearranges sentences into segmented monotonic data for simultaneous speech translation using the Large Language Model.
Outcome:	The proposed dataset improves quality and latency in siST translations by rearranging sentences into segmented monotonic data.

Towards Cross-Lingual Explanation of Artwork in Large-scale Vision Language Models (2025.findings-naacl)

Copied to clipboard

Shintaro Ozaki, Kazuki Hayashi, Yusuke Sakai, Hidetaka Kamigaito, Katsuhiko Hayashi, Taro Watanabe

Multilinguality

Challenge:	LVLMs are increasingly capable of responding in multiple languages . however, there is a lack of evaluation tools for LVLs that handle multiple languages.
Approach:	They used an extended dataset in multiple languages to evaluate LVLMs' ability to generate explanations in multiple language combinations.
Outcome:	The proposed dataset in multiple languages evaluates LVLMs' ability to generate explanations in other languages.

Simultaneous Interpretation Corpus Construction by Large Language Models in Distant Language Pair (2024.emnlp-main)

Copied to clipboard

Yusuke Sakai, Mana Makinae, Hidetaka Kamigaito, Taro Watanabe

Languages Still Left Behind: Toward a Better Multilingual Machine Translation Benchmark (2025.emnlp-main)

Challenge:	Existing siMT corpora are limited due to high costs and limited annotator capabilities.
Approach:	They propose a method to convert ST corpora into interpretation-style corpors by fine-tuning models with Large Language Models.
Outcome:	The proposed method reduces latency while achieving better quality compared to other models.

Copied to clipboard

Chihiro Taguchi, Seng Mai, Keita Kurabe, Yusuke Sakai, Georgina Agyei, Soudabeh Eslami, David Chiang

Multilinguality

Challenge:	Multilingual machine translation (MT) benchmarks are widely used to evaluate the capabilities of modern MT systems.
Approach:	They propose to use a multilingual machine translation benchmark to assess the capabilities of modern machine translation systems.
Outcome:	The FLORES+ benchmark claims to maintain a translation quality score of over 90% . however, the data in four languages falls short of the 90% quality standard .

Revisiting Compositional Generalization Capability of Large Language Models Considering Instruction Following Ability (2025.acl-long)

Copied to clipboard

Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe

IRR: Image Review Ranking Framework for Evaluating Vision-Language Models (2025.coling-main)

Challenge:	generative large language models (LLMs) compose sentences that include all given concepts but must generate sentences that adhere to the specified order.
Approach:	They propose a benchmark to evaluate compositional generalization and instruction-following abilities of generative large language models (LLMs) based on ordered coverage, which allows simultaneous evaluation of both abilities.
Outcome:	The proposed benchmark evaluates compositional generalization and instruction-following abilities of LLMs.

Copied to clipboard

Kazuki Hayashi, Kazuma Onishi, Toma Suzuki, Yusuke Ide, Seiji Gobara, Shigeki Saito, Yusuke Sakai, Hidetaka Kamigaito, Katsuhiko Hayashi, Taro Watanabe

Visual Data in NLP

Challenge:	Large-scale vision language models excel at generating factual content, but their ability to rank images from multiple perspectives has not been explored.
Approach:	They propose a framework to evaluate large-scale vision-language models by measuring their ability to rank image texts from multiple perspectives.
Outcome:	The proposed evaluation framework measures how closely LVLMs' judgments align with human interpretations.

Does Pre-trained Language Model Actually Infer Unseen Links in Knowledge Graph Completion? (2024.naacl-long)

Copied to clipboard

Yusuke Sakai, Hidetaka Kamigaito, Katsuhiko Hayashi, Taro Watanabe

IMPARA-GED: Grammatical Error Detection is Boosting Reference-free Grammatical Error Quality Estimator (2025.findings-acl)

Challenge:	Knowledge Graph Completion (KGC) is a task that infers unseen relationships between entities . traditional embedding-based methods infer missing links using only training data . a pre-trained language model (PLM)-based KGC may be ineffective in practical applications .
Approach:	They propose to use knowledge Graph Completion (KGC) to infer unseen relationships . traditional embedding-based KGC methods infer missing links only from training data . they argue that pre-trained language models acquire inference abilities through pre-training .
Outcome:	The proposed method improves performance even though it does not use memorized knowledge.

Copied to clipboard

Yusuke Sakai, Takumi Goto, Taro Watanabe

Syntactic Text Processing

Challenge:	Existing reference-free automatic grammatical error correction methods do not correlate with human evaluation.
Approach:	They propose a reference-free automatic grammatical error correction evaluation method with enhanced gramma-ed capabilities.
Outcome:	The proposed method achieves highest correlation with human evaluations on a meta-evaluation dataset.

BQA: Body Language Question Answering Dataset for Video Large Language Models (2025.acl-short)

Copied to clipboard

Shintaro Ozaki, Kazuki Hayashi, Miyu Oba, Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe

Multimodality

Challenge:	a large part of human communication relies on nonverbal cues such as facial expressions, eye contact, and body language.
Approach:	They propose to validate whether video large language models can correctly interpret body language from short clips of body language.
Outcome:	The proposed model can correctly interpret emotions from short clips of body language.

TableMBR: Minimum Bayes Risk Table Generation Based on Structural Consistency (2026.acl-srw)

Copied to clipboard

Daiki Yoshida, Hiroyuki Deguchi, Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe

Multimodality

Challenge:	Experimental results show TableMBR outperforms the baseline, achieving relative improvements of up to 15% in F1 on Rotowire and 23% in accuracy on LiveSum.
Approach:	They propose a text-to-table task that generates structured data from unstructured text . they propose 'tableMBR' that maintains structural consistency through minimum Bayes risk decoding .
Outcome:	The proposed method outperforms the baseline and achieves relative improvements in F1 and LiveSum.

Towards Artwork Explanation in Large-scale Vision Language Models (2024.acl-short)

Copied to clipboard

Kazuki Hayashi, Yusuke Sakai, Hidetaka Kamigaito, Katsuhiko Hayashi, Taro Watanabe

Explainability & Interpretability in NLP

Challenge:	Large-scale Vision-Language Models (LVLMs) output text from images and instructions, demonstrating advanced capabilities in text generation and comprehension.
Approach:	They propose to use artwork explanation generation task to quantitatively assess the understanding and utilization of artworks knowledge.
Outcome:	The proposed task evaluates the understanding and utilization of knowledge about artworks from images and titles and generates explanations using only images.

SinhalaMMLU: A Comprehensive Benchmark for Evaluating Multitask Language Understanding in Sinhala (2025.emnlp-main)

Copied to clipboard

Ashmari Pramodya, Nirasha Nelki, Heshan Shalinda, Chamila Liyanage, Yusuke Sakai, Randil Pushpananda, Ruvan Weerasinghe, Hidetaka Kamigaito, Taro Watanabe

Reliability Crisis of Reference-free Metrics for Grammatical Error Correction (2025.findings-emnlp)

Challenge:	Large Language Models (LLMs) have been evaluated mostly on global or anglocentric subjects, often neglecting low-resource languages and culturally specific content.
Approach:	They evaluate 26 Large Language Models using a multiple-choice question answering benchmark for Sinhala.
Outcome:	The new benchmarks show that Claude 3.5 sonnet and GPT-4o achieve the highest average accuracies, but overall performance remains limited.

Copied to clipboard

Takumi Goto, Yusuke Sakai, Taro Watanabe

Robustness in NLP

Challenge:	Reference-free evaluation metrics for grammatical error correction have high correlation with human judgments, but they are not designed to evaluate adversarial systems that aim to obtain unjustifiably high scores.
Approach:	They propose adversarial attack strategies for four reference-free metrics . they propose SOME, Scribendi, IMPARA, and LLM-based metrics based on these metrics a .
Outcome:	The proposed attacks outperform the current state-of-the-art for four reference-free metrics .

Simul-COMET: A Quality Metric for Simultaneous Interpretation in Distant Language Pair Considering Word Order Difference (2026.findings-acl)

Copied to clipboard

Kosuke Doi, Mana Makinae, Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe

Translating Movie Subtitles by Large Language Models using Movie-meta Information (2025.acl-srw)

Challenge:	Simultaneous interpretation (SI) uses segmenting of source speech into chunks and translating them in order.
Approach:	They propose a variation of COMET that measures monotonicity for simultaneous interpretation . they train Simul-COMET on offline translation data and show stronger alignment with evaluation scores .
Outcome:	The proposed model shows stronger alignment with evaluation scores provided by interpreters than COMET.

Copied to clipboard

Ashmari Pramodya, Yusuke Sakai, Justin Vasselli, Hidetaka Kamigaito, Taro Watanabe

Reliability of Distribution Predictions by LLMs: Insights from Counterintuitive Pseudo-Distributions (2025.naacl-srw)

Challenge:	Large language models (LLMs) have advanced natural language processing by understanding, generating, and manipulating texts.
Approach:	They propose to use movie subtitle prompts to improve translation accuracy by incorporating movie meta-information into the models.
Outcome:	The proposed prompts improve translation accuracy and reduce computational effort.

Copied to clipboard

Toma Suzuki, Ayuki Katayama, Seiji Gobara, Ryo Tsujimoto, Hibiki Nakatani, Kazuki Hayashi, Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe

mCSQA: Multilingual Commonsense Reasoning Dataset with Unified Creation Strategy by Language Models and Humans (2024.findings-acl)

Challenge:	Recent studies highlight the use of Large Language Models (LLMs) for predicting response distributions as a cost-effective survey method.
Approach:	They examine whether LLMs can rationally estimate distributions when presented with explanations that are against commonsense.
Outcome:	The proposed models can rationally estimate distributions when presented with explanations that are against commonsense, but smaller or less human-optimized models follow explanations uncritically, compared to larger models that resist counterintuitive explanations by leveraging their pretraining-acquired knowledge.

Copied to clipboard

Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe

Reasoning

Challenge:	Currently, multilingual datasets are created through translation, which cannot evaluate such language-specific aspects.
Approach:	They propose to curate a dataset for language-specific knowledge and commonsense . they propose to use multilingual commonsensiaq to leverage language models for a more efficient construction .
Outcome:	The proposed method reduces the creation cost by using multilingual LMs to create QAs . the proposed approach is based on the construction process of CSQA but with language models .

Enhancing Factuality through Consensus and Consistency in Summarization Using Minimum Bayes Risk Decoding (2026.findings-acl)

Copied to clipboard

Riza Setiawan Soetedjo, Yusuke Sakai, Hidetaka Kamigaito, Jingun Kwon, Manabu Okumura, Taro Watanabe

BannerBench: Benchmarking Vision Language Models for Multi-Ad Selection with Human Preferences (2025.findings-emnlp)

Challenge:	Existing methods to improve summarization quality are limited to using source as guidance . reranking can be effective, but there are limitations, such as relying on reference-free metrics and rely on a single metric.
Approach:	They propose a model that reranks model-generated summaries by considering consistency to the source document and consensus among the other candidates.
Outcome:	The proposed system is competitive with existing methods, with human evaluations further confirming that it is superior.

Copied to clipboard

Hiroto Otake, Peinan Zhang, Yusuke Sakai, Masato Mita, Hiroki Ouchi, Taro Watanabe