Papers by Kazuki Sakai
Cosine Similarity as Logits?: A Scalable Knowledge Probe Using Embedding Vectors from Generative Language Models (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing knowledge probes for pre-trained language models exhibit quadratic time complexity, limiting the size of knowledge graphs used for probing. |
| Approach: | They propose an embedding-based relational probe that evaluates pre-trained language models' factual knowledge retrieval capabilities. |
| Outcome: | The proposed probe achieves effective time complexity of linear order O(n), supports rank-based evaluation metrics including Hit@k, handles multi-token entity names and enables probing whilst disambiguating homographic tail-entity names. |
Creating Large-Scale Argumentation Structures for Dialogue Systems (L18-1)
Copied to clipboard
Kazuki Sakai, Akari Inago, Ryuichiro Higashinaka, Yuichiro Yoshikawa, Hiroshi Ishiguro, Junji Tomita
| Challenge: | Argumentation is a process of reaching consensus through premises and rebuttals and is important for making decisions and exchanging views. |
| Approach: | They propose to create argumentation structures in ten languages using argumentation databases . they also examine differences between the two languages to determine their effectiveness . |
| Outcome: | The proposed arguments can be applied to argumentative dialogue systems and can be used as training data. |
Diagnosing Vision Language Models’ Perception by Leveraging Human Methods for Color Vision Deficiencies (2026.eacl-long)
Copied to clipboard
| Challenge: | Large-scale Vision-Language Models (LVLMs) are being deployed in real-world settings that require visual inference. |
| Approach: | They evaluate LVLMs' ability to account for variation in color perception using the Ishihara Test. |
| Outcome: | The proposed models fail to reproduce the perceptual outcomes experienced by affected individuals and default to normative color perception. |
Towards Cross-Lingual Explanation of Artwork in Large-scale Vision Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | LVLMs are increasingly capable of responding in multiple languages . however, there is a lack of evaluation tools for LVLs that handle multiple languages. |
| Approach: | They used an extended dataset in multiple languages to evaluate LVLMs' ability to generate explanations in multiple language combinations. |
| Outcome: | The proposed dataset in multiple languages evaluates LVLMs' ability to generate explanations in other languages. |
IRR: Image Review Ranking Framework for Evaluating Vision-Language Models (2025.coling-main)
Copied to clipboard
Kazuki Hayashi, Kazuma Onishi, Toma Suzuki, Yusuke Ide, Seiji Gobara, Shigeki Saito, Yusuke Sakai, Hidetaka Kamigaito, Katsuhiko Hayashi, Taro Watanabe
| Challenge: | Large-scale vision language models excel at generating factual content, but their ability to rank images from multiple perspectives has not been explored. |
| Approach: | They propose a framework to evaluate large-scale vision-language models by measuring their ability to rank image texts from multiple perspectives. |
| Outcome: | The proposed evaluation framework measures how closely LVLMs' judgments align with human interpretations. |
BQA: Body Language Question Answering Dataset for Video Large Language Models (2025.acl-short)
Copied to clipboard
| Challenge: | a large part of human communication relies on nonverbal cues such as facial expressions, eye contact, and body language. |
| Approach: | They propose to validate whether video large language models can correctly interpret body language from short clips of body language. |
| Outcome: | The proposed model can correctly interpret emotions from short clips of body language. |
Towards Artwork Explanation in Large-scale Vision Language Models (2024.acl-short)
Copied to clipboard
| Challenge: | Large-scale Vision-Language Models (LVLMs) output text from images and instructions, demonstrating advanced capabilities in text generation and comprehension. |
| Approach: | They propose to use artwork explanation generation task to quantitatively assess the understanding and utilization of artworks knowledge. |
| Outcome: | The proposed task evaluates the understanding and utilization of knowledge about artworks from images and titles and generates explanations using only images. |
Reliability of Distribution Predictions by LLMs: Insights from Counterintuitive Pseudo-Distributions (2025.naacl-srw)
Copied to clipboard
Toma Suzuki, Ayuki Katayama, Seiji Gobara, Ryo Tsujimoto, Hibiki Nakatani, Kazuki Hayashi, Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe
| Challenge: | Recent studies highlight the use of Large Language Models (LLMs) for predicting response distributions as a cost-effective survey method. |
| Approach: | They examine whether LLMs can rationally estimate distributions when presented with explanations that are against commonsense. |
| Outcome: | The proposed models can rationally estimate distributions when presented with explanations that are against commonsense, but smaller or less human-optimized models follow explanations uncritically, compared to larger models that resist counterintuitive explanations by leveraging their pretraining-acquired knowledge. |
LoCt-Instruct: An Automatic Pipeline for Constructing Datasets of Logical Continuous Instructions (2025.emnlp-main)
Copied to clipboard
Hongyu Sun, Yusuke Sakai, Haruki Sakajo, Shintaro Ozaki, Kazuki Hayashi, Hidetaka Kamigaito, Taro Watanabe
| Challenge: | Existing instruction following datasets lack logical coherence across turns, narrow topical breadth and heavy manual effort. |
| Approach: | They propose a pipeline that leverages LLMs’ reasoning capabilities to assemble rich, topic-related single-instruction data into multi-turn dialogues and produce chains that are logically coherent, progressively deepen in content, and span diverse domains without fixed templates or extensive human annotation. |
| Outcome: | The proposed pipeline improves the performance of existing LLMs by integrating multiple topic-related data into multi-turn dialogues without fixed templates or extensive human annotation. |