Papers by Piotr Zelasko
VoiceTextBlender: Augmenting Large Language Models with Speech Capabilities via Single-Stage Joint Speech-Text Supervised Fine-Tuning (2025.naacl-long)
Copied to clipboard
Yifan Peng, Krishna C Puvvada, Zhehuai Chen, Piotr Zelasko, He Huang, Kunal Dhawan, Ke Hu, Shinji Watanabe, Jagadeesh Balam, Boris Ginsburg
| Challenge: | Recent studies have augmented large language models (LLMs) with speech capabilities, leading to the development of speech language models. |
| Approach: | They propose a single-stage joint speech-text SFT approach for training SpeechLMs . their model combines text-only SFT data with three types of speech-related data . |
| Outcome: | The proposed model outperforms previous SpeechLMs on speech-based QA tasks while maintaining original speech-only capabilities. |
NeKo: Cross-Modality Post-Recognition Error Correction with Tasks-Guided Mixture-of-Experts Language Model (2025.acl-industry)
Copied to clipboard
Yen-Ting Lin, Zhehuai Chen, Piotr Zelasko, Zhen Wan, Xuesong Yang, Zih-Ching Chen, Krishna C Puvvada, Ke Hu, Szu-Wei Fu, Jun Wei Chiu, Jagadeesh Balam, Boris Ginsburg, Yu-Chiang Frank Wang, Chao-Han Huck Yang
| Challenge: | Existing methods to train a model on a mixture of domain datasets require separate correction language models. |
| Approach: | They propose a multi-task correction MoE that trains experts to become an "expert" of speech-to-text, language-totext and vision-to text datasets by learning to route each dataset’s tokens to its mapped expert. |
| Outcome: | The proposed model outperforms GPT-3.5 and Claude-3.5-Sonnet on the Open ASR Leaderboard and reaches an average relative 5.0% WER reduction and substantial improvements in BLEU scores. |