Papers by Shinji Watanabe

Optimizing Conversational Quality in Spoken Dialogue Systems with Reinforcement Learning from AI Feedback (2026.findings-acl)

Challenge:	Existing evaluation methods for transfer learning are limited in speech research . authors show that pre-trained models transfer well across multiple tasks .
Approach:	They propose a benchmark to evaluate pre-trained models by increasing task diversity and difficulty over SUPERB.
Outcome:	The proposed benchmark increases task diversity and difficulty over SUPERB-SG.

Copied to clipboard

Siddhant Arora, Jinchuan Tian, Jiatong Shi, Hayato Futami, Yosuke Kashiwagi, Emiru Tsunoo, Shinji Watanabe

VoiceTextBlender: Augmenting Large Language Models with Speech Capabilities via Single-Stage Joint Speech-Text Supervised Fine-Tuning (2025.naacl-long)

Challenge:	Existing studies on reinforcement learning from human or AI feedback have focused on semantic rewards at the utterance level.
Approach:	They propose a multi-reward RLAIF framework for speech-in/speech-out dialogue systems . they combine semantic, audio-quality, and emotion-consistency rewards .
Outcome:	The proposed framework improves speech-in/speech-out dialogue system quality . it combines semantic, audio-quality, and emotion-consistency rewards . the proposed framework is available to download from the cdc.

Copied to clipboard

Yifan Peng, Krishna C Puvvada, Zhehuai Chen, Piotr Zelasko, He Huang, Kunal Dhawan, Ke Hu, Shinji Watanabe, Jagadeesh Balam, Boris Ginsburg

UniverSLU: Universal Spoken Language Understanding for Diverse Tasks with Natural Language Instructions (2024.naacl-long)

Challenge:	Recent studies have augmented large language models (LLMs) with speech capabilities, leading to the development of speech language models.
Approach:	They propose a single-stage joint speech-text SFT approach for training SpeechLMs . their model combines text-only SFT data with three types of speech-related data .
Outcome:	The proposed model outperforms previous SpeechLMs on speech-based QA tasks while maintaining original speech-only capabilities.

Copied to clipboard

Siddhant Arora, Hayato Futami, Jee-weon Jung, Yifan Peng, Roshan Sharma, Yosuke Kashiwagi, Emiru Tsunoo, Karen Livescu, Shinji Watanabe

Low-Resource NLP

Challenge:	Recent studies leverage large language models with multi-tasking capabilities, using natural language prompts to guide the model’s behavior and surpassing performance of task-specific models.
Approach:	They adapt a pre-trained automatic speech recognition model to additional tasks using single-token task specifiers.
Outcome:	The proposed model can generalize to new datasets and languages for seen task types.

Summarizing Speech: A Comprehensive Survey (2025.emnlp-main)

Copied to clipboard

Fabian Retkowski, Maike Züfle, Andreas Sudmann, Dinah Pfau, Shinji Watanabe, Jan Niehues, Alexander Waibel

Full-Duplex-Bench-v2: A Multi-Turn Evaluation Framework for Duplex Dialogue Systems with an Automated Examiner (2026.acl-short)

Challenge:	Podcasts and other audiovisual content are becoming more and more a part of everyday communication and the digital age is changing from text to voice.
Approach:	They synthesize the current state of the field and highlight the need for realistic evaluation benchmarks and multilingual datasets.
Outcome:	The proposed frameworks are based on evaluation protocols and datasets and highlight the need for realistic benchmarks and multilingual datasets.

Copied to clipboard

Guan-Ting Lin, Shih-Yun Shan Kuan, Jiatong Shi, Kai-Wei Chang, Siddhant Arora, Shinji Watanabe, Hung-yi Lee

Dialogue Systems & Conversational Agents

Challenge:	Full-duplex speech agents are often half-duplice, alternating turns between user and system.
Approach:	They propose a streaming framework that integrates with an examiner that enforces staged goals under two pacing setups.
Outcome:	The framework reports fluency, multi-turn instruction following, and task-specific competence.

Wav2Gloss: Generating Interlinear Glossed Text from Speech (2024.acl-long)

Copied to clipboard

Taiqi He, Kwanghee Choi, Lindia Tjuatja, Nathaniel Robinson, Jiatong Shi, Shinji Watanabe, Graham Neubig, David Mortensen, Lori Levin

ESPnet-ST: All-in-One Speech Translation Toolkit (2020.acl-demos)

Challenge:	Interlinear Glossed Text (IGT) is a form of linguistic annotation that can support documentation and resource creation for endangered languages.
Approach:	They propose a task in which these four annotation components are extracted automatically from speech and introduce a dataset to lay the groundwork for future research on IGT generation from speech.
Outcome:	The proposed dataset provides the first dataset to lay the groundwork for future research on IGT generation from speech, including end-to-end versus cascaded, monolingual versus multilingual, and single-task versus multiple-task approaches.

Copied to clipboard

Hirofumi Inaguma, Shun Kiyono, Kevin Duh, Shigeki Karita, Nelson Yalta, Tomoki Hayashi, Shinji Watanabe

Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translation (2021.naacl-main)

Challenge:	ESPnet-ST is a new project for the quick development of speech-to-speech translation systems.
Approach:	They propose a framework for rapid development of speech-to-speech translation systems . they provide all-in-one recipes including data pre-processing, feature extraction, training, and decoding pipelines .
Outcome:	The proposed model outperforms the current state-of-the-art models on a wide range of benchmark datasets.

Copied to clipboard

Hirofumi Inaguma, Tatsuya Kawahara, Shinji Watanabe

POWSM: A Phonetic Open Whisper-Style Speech Foundation Model (2026.acl-long)

Challenge:	End-to-end speech translation models can be trained to leverage source text . however, since the input modalities are different, it is difficult to leverage the source text successfully.
Approach:	They propose to leverage source transcriptions via pre-training and joint training with ASR and NMT tasks.
Outcome:	The proposed model predicts paraphrased transcriptions as an auxiliary task with a single decoder.

Copied to clipboard

Chin-Jou Li, Kalvin Chang, Shikhar Bharadwaj, Eunjung Yeo, Kwanghee Choi, Jian Zhu, David R. Mortensen, Shinji Watanabe

Leveraging End-to-End ASR for Endangered Language Documentation: An Empirical Study on Yolóxochitl Mixtec (2021.eacl-main)

Challenge:	Phone-level modeling of speech is a common approach to speech recognition, but it relies on task-specific architectures and datasets.
Approach:	They propose a phonetic framework capable of performing multiple phone-related tasks . they propose 'Phonetic Open Whisper-style Speech Model' that can perform these tasks together .
Outcome:	The proposed model outperforms or matches specialized PR models of similar size while supporting G2P, P2G, and ASR.

Copied to clipboard

Jiatong Shi, Jonathan D. Amith, Rey Castillo García, Esteban Guadalupe Sierra, Kevin Duh, Shinji Watanabe

ESPnet-ST-v2: Multipurpose Spoken Language Translation Toolkit (2023.acl-demo)

Challenge:	End-to-end ASR systems that eschew linguistic resources but are more dependent on large-data settings are suggested as a solution to EL documentation bottlenecks.
Approach:	They propose to build an end-to-end ASR system that is reproducible by the ASR community and propose a novice transcription correction task.
Outcome:	The proposed method would mitigate bottlenecks and shortages in transcribers . it is based on a Yoloxóchitl Mixtec corpus and is reproducible by the ASR community.

Copied to clipboard

Brian Yan, Jiatong Shi, Yun Tang, Hirofumi Inaguma, Yifan Peng, Siddharth Dalmia, Peter Polák, Patrick Fernandes, Dan Berrebbi, Tomoki Hayashi, Xiaohui Zhang, Zhaoheng Ni, Moto Hira, Soumi Maiti, Juan Pino, Shinji Watanabe

UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Units (2023.acl-long)

Challenge:	ESPnet-ST-v2 is a revamp of the open-source spoken language translation toolkit . it supports offline speech-to-text translation (ST), simultaneous speech- to-text (SST), and offline speech to-speech (S2ST)
Approach:	They propose to revamp the open-source ESPnet-ST toolkit to support offline speech-to-text translation, simultaneous speech- to-text and offline speech to-speech translation.
Outcome:	The updated version of ESPnet-ST supports offline speech-to-text translation (ST), simultaneous speech- to-text (SST), and offline speech to-speech translation (S2ST).

Copied to clipboard

Hirofumi Inaguma, Sravya Popuri, Ilia Kulikov, Peng-Jen Chen, Changhan Wang, Yu-An Chung, Yun Tang, Ann Lee, Shinji Watanabe, Juan Pino

Speech-Hands: A Self-Reflection Voice Agentic Approach to Speech Recognition and Audio Reasoning with Omni Perception (2026.acl-long)

Challenge:	Experimental evaluations show that UnitY outperforms a single-pass speech-to-unit translation model by 2.5-4.2 ASR-BLEU with 2.83x decoding speed-up.
Approach:	They propose a two-pass direct S2ST architecture which generates textual representations and predicts discrete acoustic units . they show that UnitY outperforms a single-pass speech-to-unit translation model by 2.5-4.2 ASR-BLEU with 2.83x decoding speed-up.
Outcome:	The proposed architecture outperforms a single-pass speech-to-unit translation model by 2.5-4.2 ASR-BLEU with 2.83x decoding speed-up on large datasets.

Copied to clipboard

Zhen Wan, Chao-Han Huck Yang, Jinchuan Tian, Hanrong Ye, Ankita Pasad, Szu-Wei Fu, Arushi Goel, Ryo Hachiuma, Shizhe Diao, Kunal Dhawan, Sreyan Ghosh, Yusuke Hirota, Zhehuai Chen, Rafael Valle, Chenhui Chu, Shinji Watanabe, Boris Ginsburg, Yu-Chiang Frank Wang

SpeechIQ: Speech-Agentic Intelligence Quotient Across Cognitive Levels in Voice Understanding by Large Language Models (2025.acl-long)

Challenge:	naively fine-tuning an omni-model on speech recognition and external sound understanding tasks often degrades performance . Xie and Wu's framework, Speech-Hands, recasts the problem as an explicit self-reflection decision.
Approach:	They propose a voice-agentic framework that learns one critical omni-understanding skill: trusting itself versus external audio perception.
Outcome:	The proposed framework outperforms baseline models on the OpenASR leaderboard by 12.1% WER and high F1 on audio QA decisions.

Copied to clipboard

Zhen Wan, Chao-Han Huck Yang, Yahan Yu, Jinchuan Tian, Sheng Li, Ke Hu, Zhehuai Chen, Shinji Watanabe, Fei Cheng, Chenhui Chu, Sadao Kurohashi

Massively Multilingual Adversarial Speech Recognition (N19-1)

Challenge:	SIQ quantifies voice understanding abilities and provides unified comparisons between cascaded methods and end-to-end models.
Approach:	They propose a human cognition-inspired evaluation pipeline for voice understanding large language models (LLM_Voice) that quantifies voice understanding abilities and provides unified comparisons between cascaded methods and end-to-end models.
Outcome:	The proposed framework quantifies voice understanding abilities and provides unified comparisons between cascaded methods and end-to-end models, identifies annotation errors in existing benchmarks, and detects hallucinations in LLM_Voice.

Copied to clipboard

Oliver Adams, Matthew Wiesner, Shinji Watanabe, David Yarowsky

PRiSM: Benchmarking Phone Realization in Speech Models (2026.acl-long)

Challenge:	Prior work in multilingual and cross-lingual speech recognition has been limited to a subset of the world's most-spoken languages.
Approach:	They propose to use phonemes and phonemes as pretraining objectives to encourage language-independent representations.
Outcome:	The proposed model is able to learn language-independent representations of speech using multilingual training.

Copied to clipboard

Shikhar Bharadwaj, Chin-Jou Li, Yoonjae Kim, Kwanghee Choi, Eunjung Yeo, Ryan Soh-Eun Shim, Hanyu Zhou, Brendon Boldt, Karen Rosero, Kalvin Chang, Darsh Agrawal, Keer Xu, Chao-Han Huck Yang, Jian Zhu, Shinji Watanabe, David R. Mortensen

PlanRAG-Audio: Planning and Retrieval Augmented Generation for Long-form Audio Understanding (2026.findings-acl)

Challenge:	Existing evaluations of phone recognition systems only measure surface-level transcription accuracy.
Approach:	They propose to standardize transcription-based evaluation and assess downstream utility in clinical, educational, and multilingual settings with transcription and representation probes.
Outcome:	The proposed system outperforms LALMs in clinical, educational, and multilingual settings.

Copied to clipboard

Masao Someki, Chien-yu Huang, Siddhant Arora, Samuele Cornell, Markus Müller, Nathan Susanj, Rupak Vignesh Swaminathan, Grant Strimel, Jing Liu, Shinji Watanabe

End-to-end ASR to jointly predict transcriptions and linguistic annotations (2021.naacl-main)

Challenge:	Long-form audio understanding poses significant challenges due to the extreme length of audio sequences and the need to reason over heterogeneous acoustic cues distributed over time.
Approach:	They propose a retrieval-augmented generation framework for scalable long-form audio understanding . planRAG-Audio explicitly plans which modalities and temporal spans are required for a given query .
Outcome:	Experiments show that planRAG-Audio reduces the length of inputs for long-form audio models . the proposed framework can efficiently reason over long-term speech data .

Copied to clipboard

Motoi Omachi, Yuya Fujita, Shinji Watanabe, Matthew Wiesner

Leveraging Allophony in Self-Supervised Speech Models for Atypical Pronunciation Assessment (2025.naacl-long)

Challenge:	Existing models generate audio transcripts by sequentially producing likely graphemes, or multi-graphemic units, from which lexical items of a language can be recovered.
Approach:	They propose a Transformer-based sequence-to-sequence model for automatic speech recognition that can produce high-quality transcriptions and linguistic annotations.
Outcome:	The proposed model can produce high-quality transcriptions and linguistic annotations on Japanese and English audio datasets.

Copied to clipboard

Kwanghee Choi, Eunjung Yeo, Kalvin Chang, Shinji Watanabe, David R Mortensen

BSCodec: A Band-Split Neural Codec for High-Quality Universal Audio Reconstruction (2026.findings-eacl)

Challenge:	Recent phoneme classifiers treat allophonic variation as a single phoneme . atypical pronunciation assessment requires distinguishing between a typical and asymmetric pronunciations .
Approach:	They propose a new approach that leverages Gaussian mixture models to model phoneme distributions with multiple subclusters.
Outcome:	The proposed approach achieves state-of-the-art across dysarthric and non-native speech datasets.

Copied to clipboard

Haoran Wang, Jiatong Shi, Jinchuan Tian, Bohan Li, Kai Yu, Shinji Watanabe

FastAdaSP: Multitask-Adapted Efficient Inference for Large Speech Language Model (2024.emnlp-industry)

Challenge:	Neural audio codecs have enabled high-fidelity reconstruction of speech, music and sound . however, speech-optimized codec systems suffer degradation on music or sound if they ignore spectral differences .
Approach:	They propose a neural audio codec that splits the spectral dimension into separate bands and compresses each band independently.
Outcome:	Experimental results show that BSCodec achieves better reconstruction quality on music and sound compared to existing codecs.

Copied to clipboard

Yichen Lu, Jiaqi Song, Chao-Han Yang, Shinji Watanabe

Semantic Text Processing

Challenge:	Unlike other modalities, speech has unique temporal dependencies, making efficient inference methods unexplored.
Approach:	They propose a weighted token merging framework specifically designed for speech-related tasks to improve the trade-off between efficiency and performance.
Outcome:	The proposed method achieves state-of-the-art efficiency-performance trade-off on speech-related tasks.

CTC Alignments Improve Autoregressive Translation (2023.eacl-main)

Copied to clipboard

Brian Yan, Siddharth Dalmia, Yosuke Higuchi, Graham Neubig, Florian Metze, Alan W Black, Shinji Watanabe

A Purely End-to-End System for Multi-speaker Speech Recognition (P18-1)

Challenge:	Connectionist Temporal Classification (CTC) is widely used for automatic speech recognition (ASR) but lags behind attentional decoder approaches in terms of translation quality.
Approach:	They propose to use a CTC/attention framework to validate this hypothesis by modifying the Hybrid CTC-Attention model proposed for automatic speech recognition to support text-to-text translation (MT) and speech-totext translation.
Outcome:	The proposed model outperforms pure-attention baselines across six translation tasks.

Copied to clipboard

Hiroshi Seki, Takaaki Hori, Shinji Watanabe, Jonathan Le Roux, John R. Hershey

Towards Robust Speech Representation Learning for Thousands of Languages (2024.emnlp-main)

Challenge:	Existing methods for multi-speaker speech recognition require isolated source signals or senone alignments for effective learning.
Approach:	They propose a sequence-to-sequence framework to decode multiple label sequences from a single speech sequence by unifying source separation and speech recognition functions in an end-to end manner.
Outcome:	The proposed model improves on existing models by 83.1% relative to previous models with explicit separation and recognition modules.

Copied to clipboard

William Chen, Wangyou Zhang, Yifan Peng, Xinjian Li, Jinchuan Tian, Jiatong Shi, Xuankai Chang, Soumi Maiti, Karen Livescu, Shinji Watanabe

Robustness in NLP

Challenge:	XEUS is a cross-lingual encoder for universal speech that can be trained on 1 million hours of data across 4057 languages.
Approach:	They propose a Cross-lingual Encoder for Universal Speech that can be trained on 1 million hours of data across 4057 languages and a newly created corpus of 7400+ hours from 4057 .
Outcome:	The proposed model outperforms state-of-the-art models on several benchmarks and outperfies MMS 1B and w2v-BERT 2.0 v2 by 0.8% and 4.4% respectively.

Self-supervised Representation Learning for Speech Processing (2022.naacl-tutorials)

Copied to clipboard

Hung-yi Lee, Abdelrahman Mohamed, Shinji Watanabe, Tara Sainath, Karen Livescu, Shang-Wen Li, Shu-wen Yang, Katrin Kirchhoff

Semantic Text Processing

Challenge:	Self-supervised representation learning (SSL) uses proxy supervised learning tasks to obtain training data from unlabeled corpora.
Approach:	They propose to survey the latest SSL techniques, tools, datasets, and performance achievement in speech processing to scale up current machine learning technologies.
Outcome:	The proposed tutorial is highly relevant to the special theme of ACL about language diversity.

Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Tasks (2021.naacl-main)

Copied to clipboard

Siddharth Dalmia, Brian Yan, Vikas Raunak, Florian Metze, Shinji Watanabe

Phone Inventories and Recognition for Every Language (2022.lrec-1)

Challenge:	ESPnet framework exploits compositionality to learn searchable hidden representations at intermediate stages of a sequence model using decomposed sub-tasks.
Approach:	They propose a framework that exploits compositionality to learn searchable hidden representations at intermediate stages of a sequence model using decomposed sub-tasks.
Outcome:	The proposed framework outperforms the state-of-the-art on speech translation tasks by +6 and +3 BLEU on the two test sets of Fisher-CallHome and +4 BLUE on the English-German and English-French test sets.

Copied to clipboard

Xinjian Li, Florian Metze, David R. Mortensen, Alan W Black, Shinji Watanabe

Syntactic Text Processing

Challenge:	Identifying phone inventories is crucial component in language documentation and preservation of endangered languages.
Approach:	They propose a probabilistic and non-probabilistic phone inventory model that estimates the phone inventory for any language listed in Glottolog.
Outcome:	The proposed model outperforms baseline models by 6.5 F1 and improves the PER (phone error rate) in phone recognition by 25%.

OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identification (2024.acl-long)

Copied to clipboard

Yifan Peng, Yui Sudo, Muhammad Shakeel, Shinji Watanabe

SLUE Phase-2: A Benchmark Suite of Diverse Spoken Language Understanding Tasks (2023.acl-long)

Challenge:	Autoregressive models can be slower during inference and have potential risks of hallucination.
Approach:	They propose an encoder-only speech foundation model based on Connectionist Temporal Classification.
Outcome:	The proposed model improves on 180k hours of public audio data for multilingual speech recognition, speech translation, and language identification.

Copied to clipboard

Suwon Shon, Siddhant Arora, Chyi-Jiunn Lin, Ankita Pasad, Felix Wu, Roshan S Sharma, Wei-Lun Wu, Hung-yi Lee, Karen Livescu, Shinji Watanabe

BERT Meets CTC: New Formulation of End-to-End Speech Recognition with Pre-trained Masked Language Model (2022.findings-emnlp)

Challenge:	Spoken language understanding (SLU) tasks have received little attention and resources compared to lower-level tasks like speech and speaker recognition.
Approach:	They propose annotated SLU benchmark tasks based on freely available speech data to complement existing benchmarks and address gaps in the evaluation landscape.
Outcome:	The proposed benchmarks complement existing benchmarks and address gaps in the evaluation landscape.

Copied to clipboard

Yosuke Higuchi, Brian Yan, Siddhant Arora, Tetsuji Ogawa, Tetsunori Kobayashi, Shinji Watanabe

Language Models

Challenge:	Existing approaches to connectionist temporal classification (CTC) are based on pre-trained language models (LMs)
Approach:	They propose a formulation of connectionist temporal classification that relaxes the conditional independence assumptions used in conventional CTC and incorporates linguistic knowledge through explicit output dependency.
Outcome:	The proposed model improves over conventional approaches across variations in speaking styles and languages while maintaining CTC’s training efficiency.

CSPB: Conversational Speech Processing Benchmark for Self-supervised Speech Models (2026.eacl-long)

Copied to clipboard

Zili Huang, Matthew Maciejewski, Leibny Paola Garcia Perera, Shinji Watanabe, Sanjeev Khudanpur

Responsible & Trustworthy NLP

Challenge:	Existing benchmarks focus on clean, single-speaker, single channel audio, failing to reflect the complexities of natural human interaction.
Approach:	They propose a benchmark to assess the robustness of self-supervised speech models in conversational settings.
Outcome:	The proposed benchmark assesses the robustness of self-supervised speech models in conversational scenarios.

On the Evaluation of Speech Foundation Models for Spoken Language Understanding (2024.findings-acl)

Copied to clipboard

Siddhant Arora, Ankita Pasad, Chung-Ming Chien, Jionghao Han, Roshan Sharma, Jee-weon Jung, Hira Dhamyal, William Chen, Suwon Shon, Hung-yi Lee, Karen Livescu, Shinji Watanabe

Token-level Sequence Labeling for Spoken Language Understanding using Compositional End-to-End Models (2022.findings-emnlp)

Challenge:	Spoken language understanding evaluation (SLUE) benchmarks are used to benchmark complex spoken language understanding tasks on natural speech.
Approach:	They propose a set of benchmark tasks to evaluate spoken language understanding on natural speech . they use pre-trained speech foundation models to evaluate the utility of different SFMs .
Outcome:	The proposed framework outperforms pre-trained speech foundation models on natural speech . the proposed framework also outperformed self-supervised SFMs on the sequence generation tasks .

Copied to clipboard

Siddhant Arora, Siddharth Dalmia, Brian Yan, Florian Metze, Alan W Black, Shinji Watanabe