Papers by Shizhe Chen
InfoMetIC: An Informative Metric for Reference-free Image Caption Evaluation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing image captioning metrics provide a single score to measure caption qualities, which are less explainable and informative. |
| Approach: | They propose an Informative Metric for Reference-free Image Caption evaluation to support this feedback . they propose to provide a text precision score, a vision recall score and an overall quality score . |
| Outcome: | The proposed method improves on existing metrics on multiple benchmarks and compares coarse-grained scores with human judgements. |
On the Difference of BERT-style and CLIP-style Text Encoders (2023.findings-acl)
Copied to clipboard
| Challenge: | Masked language modeling is one of the most popular pretraining recipes in natural language processing. |
| Approach: | They analyze BERT-style and CLIP-style text encoders from three experiments . they show that CLIP style encoder is equipped with synesthesia for the cross-modal association . |
| Outcome: | The proposed models outperform BERT-style models on vision-centric text understanding tasks, but have synesthesia for the cross-modal association, similar to the senses of humans. |
Speech-Hands: A Self-Reflection Voice Agentic Approach to Speech Recognition and Audio Reasoning with Omni Perception (2026.acl-long)
Copied to clipboard
Zhen Wan, Chao-Han Huck Yang, Jinchuan Tian, Hanrong Ye, Ankita Pasad, Szu-Wei Fu, Arushi Goel, Ryo Hachiuma, Shizhe Diao, Kunal Dhawan, Sreyan Ghosh, Yusuke Hirota, Zhehuai Chen, Rafael Valle, Chenhui Chu, Shinji Watanabe, Boris Ginsburg, Yu-Chiang Frank Wang
| Challenge: | naively fine-tuning an omni-model on speech recognition and external sound understanding tasks often degrades performance . Xie and Wu's framework, Speech-Hands, recasts the problem as an explicit self-reflection decision. |
| Approach: | They propose a voice-agentic framework that learns one critical omni-understanding skill: trusting itself versus external audio perception. |
| Outcome: | The proposed framework outperforms baseline models on the OpenASR leaderboard by 12.1% WER and high F1 on audio QA decisions. |
R-Tuning: Instructing Large Language Models to Say ‘I Don’t Know’ (2024.naacl-long)
Copied to clipboard
Hanning Zhang, Shizhe Diao, Yong Lin, Yi Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji, Tong Zhang
| Challenge: | Existing methods for instruction tuning force the model to complete a sentence no matter whether it knows the knowledge or not. |
| Approach: | They propose a new approach to tuning large language models to refrain from answering questions beyond its parametric knowledge by identifying the disparity in parametric and parametric information. |
| Outcome: | The proposed approach improves a model’s ability to answer known questions and refrain from answering unknown questions. |
SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationales (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to elicit confidence from large language models are limited to binary or inaccurate group-level confidence estimates. |
| Approach: | They propose a training framework that teaches LLMs to express more fine-grained confidence estimates. |
| Outcome: | The proposed training framework reduces the confidence calibration error and maintains the performance of the model. |
YouMakeup: A Large-Scale Domain-Specific Multimodal Dataset for Fine-Grained Semantic Comprehension (D19-1)
Copied to clipboard
| Challenge: | Multimodal semantic comprehension has attracted increasing research interest recently such as visual question answering and caption generation. |
| Approach: | They propose to use a large-scale multimodal instructional video dataset to support fine-grained comprehension research in specific domain. |
| Outcome: | The proposed dataset contains 2,800 videos from YouTube, spanning more than 420 hours in total. |
FIRST: Teach A Reliable Large Language Model Through Efficient Trustworthy Distillation (2024.emnlp-main)
Copied to clipboard
KaShun Shum, Minrui Xu, Jianshu Zhang, Zixin Chen, Shizhe Diao, Hanze Dong, Jipeng Zhang, Muhammad Raza
| Challenge: | Experimental results show that a well-calibrated model is more reliable than a fine-tuned model due to “tuning-induced mis-calibration”. |
| Approach: | They propose a method which utilizes a small portion of teacher’s knowledge to obtain a reliable language model in a cost-efficient way. |
| Outcome: | The proposed method reduces the computational burden by utilizing teacher's knowledge to obtain a reliable language model in a cost-efficient way. |
MuKA: Multimodal Knowledge Augmented Visual Information-Seeking (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods for visual information-seeking tasks rely on textual knowledge . existing methods can impair information retrieval and confuse MLLMs . |
| Approach: | They propose a framework which leverages a multimodal knowledge base to address these limitations. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on the InfoSeek and E-VQA benchmarks. |