Papers by Kunal Dhawan
VoiceTextBlender: Augmenting Large Language Models with Speech Capabilities via Single-Stage Joint Speech-Text Supervised Fine-Tuning (2025.naacl-long)
Copied to clipboard
Yifan Peng, Krishna C Puvvada, Zhehuai Chen, Piotr Zelasko, He Huang, Kunal Dhawan, Ke Hu, Shinji Watanabe, Jagadeesh Balam, Boris Ginsburg
| Challenge: | Recent studies have augmented large language models (LLMs) with speech capabilities, leading to the development of speech language models. |
| Approach: | They propose a single-stage joint speech-text SFT approach for training SpeechLMs . their model combines text-only SFT data with three types of speech-related data . |
| Outcome: | The proposed model outperforms previous SpeechLMs on speech-based QA tasks while maintaining original speech-only capabilities. |
Speech-Hands: A Self-Reflection Voice Agentic Approach to Speech Recognition and Audio Reasoning with Omni Perception (2026.acl-long)
Copied to clipboard
Zhen Wan, Chao-Han Huck Yang, Jinchuan Tian, Hanrong Ye, Ankita Pasad, Szu-Wei Fu, Arushi Goel, Ryo Hachiuma, Shizhe Diao, Kunal Dhawan, Sreyan Ghosh, Yusuke Hirota, Zhehuai Chen, Rafael Valle, Chenhui Chu, Shinji Watanabe, Boris Ginsburg, Yu-Chiang Frank Wang
| Challenge: | naively fine-tuning an omni-model on speech recognition and external sound understanding tasks often degrades performance . Xie and Wu's framework, Speech-Hands, recasts the problem as an explicit self-reflection decision. |
| Approach: | They propose a voice-agentic framework that learns one critical omni-understanding skill: trusting itself versus external audio perception. |
| Outcome: | The proposed framework outperforms baseline models on the OpenASR leaderboard by 12.1% WER and high F1 on audio QA decisions. |