Whisper-UT: A Unified Translation Framework for Speech and Text (2025.emnlp-main)
Copied to clipboard
Cihan Xiao, Matthew Wiesner, Debashish Chakraborty, Reno Kriz, Keith Cunningham, Kenton Murray, Kevin Duh, Luis Tavarez-Arce, Paul McNamee, Sanjeev Khudanpur
| Challenge: | Encoder-decoder models have achieved remarkable success in speech and text tasks, but efficiently adapting them to diverse uni/multimodal scenarios remains a challenge. |
| Approach: | They propose a framework that leverages lightweight adapters to enable seamless adaptation across tasks. |
| Outcome: | The proposed framework improves speech translation performance through a 2-stage decoding strategy without requiring 3-way parallel data. |
Similar Papers
SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing (2022.acl-long)
Copied to clipboard
Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, Zhihua Wei, Yao Qian, Jinyu Li, Furu Wei
| Challenge: | Existing work shows that pre-trained models can improve in various natural language processing tasks. |
| Approach: | They propose a unified-modal encoder-decoder framework that pre-trains speech-text representations using large-scale unlabeled speech and text data. |
| Outcome: | The proposed framework is superior to existing models on speech-to-text processing tasks. |
S2ST-Omni: Hierarchical Language-Aware SpeechLLM Adaptation for Multilingual Speech-to-Speech Translation (2026.findings-acl)
Copied to clipboard
| Challenge: | S2ST-Omni integrates a speech-to-text frontend with a modular, plug-and-play text-tospeech backend. |
| Approach: | They propose a compositional S2ST framework that integrates a speech-to-text frontend with a modular, plug-and-play text-tospeech backend. |
| Outcome: | The proposed framework outperforms existing frameworks in translation and synthesis . it integrates a speech-to-text translation frontend with a plug-and-play text-tospeech backend . |
Cross-Lingual Transfer Learning for Speech Translation (2025.naacl-short)
Copied to clipboard
| Challenge: | Increasing interest in building multilingual foundation models for NLP and speech research has led to limited data collection for training ST systems. |
| Approach: | They propose to use Whisper to explore the behavior of multilingual speech foundation models with restricted data. |
| Outcome: | The proposed model can translate to Chinese with a single language, and it can perform transcriptions in other languages. |
Tied Multitask Learning for Neural Speech Translation (N18-1)
Copied to clipboard
| Challenge: | Recent efforts in endangered language documentation focus on collecting spoken language resources . BULB project uses mobile app to collect spoken resources accompanied by spoken translations . |
| Approach: | They propose a model where the second task decoder receives information from the first task . they apply regularization that encourages transitivity and invertibility to the model . |
| Outcome: | The proposed model improves performance on low-resource speech transcription and translation tasks. |
Leveraging Large Pre-trained Multilingual Models for High-Quality Speech-to-Text Translation on Industry Scenarios (2025.coling-main)
Copied to clipboard
| Challenge: | Speech-to-Text Translation systems rely on a sequential pipeline that combines ASR and MT models. |
| Approach: | They propose a parameter-efficient framework that integrates one LPSM with a multilingual MT engine. |
| Outcome: | The proposed framework integrates one LPSM with a multilingual MT engine. |
Unified Speech-Text Pre-training for Speech Translation and Recognition (2022.acl-long)
Copied to clipboard
Yun Tang, Hongyu Gong, Ning Dong, Changhan Wang, Wei-Ning Hsu, Jiatao Gu, Alexei Baevski, Xian Li, Abdelrahman Mohamed, Michael Auli, Juan Pino
| Challenge: | Existing methods to pre-train speech and text use unlabeled data to learn universal feature representations. |
| Approach: | They propose a method to jointly pre-train speech and text in an encoder-decoder modeling framework for speech translation and recognition. |
| Outcome: | The proposed method achieves between 1.7 and 2.3 BLEU improvement above the state of the art on the MuST-C speech translation dataset and comparable WERs to wav2vec 2.0 on the Librispeech speech recognition task. |
WavLLM: Towards Robust and Adaptive Speech Large Language Model (2024.findings-emnlp)
Copied to clipboard
Shujie Hu, Long Zhou, Shujie Liu, Sanyuan Chen, Lingwei Meng, Hongkun Hao, Jing Pan, Xunying Liu, Jinyu Li, Sunit Sivasankaran, Linquan Liu, Furu Wei
| Challenge: | Recent advances in large language models (LLMs) have expanded their scope to encompass multimodal functions. |
| Approach: | They propose a robust and adaptive speech large language model with dual encoders . they validate the model on universal speech benchmarks and apply it to specialized speech-question-answer datasets based on a CoT approach . |
| Outcome: | The proposed model achieves state-of-the-art performance across a range of speech tasks on the same model size. |
A Simple and Effective Unified Encoder for Document-Level Machine Translation (2020.acl-main)
Copied to clipboard
| Challenge: | Existing models for document-level machine translation use two separate encoders to model the source sentences and document- level contexts. |
| Approach: | They propose a unified encoder that can outperform existing models of dual-encoder models . they propose to use document-level contexts to model the interaction between the contexts and the source sentences . |
| Outcome: | The proposed model outperforms baseline models of dual-encoder models in terms of BLEU and METEOR scores. |
AdaST: Dynamically Adapting Encoder States in the Decoder for End-to-End Speech-to-Text Translation (2021.findings-acl)
Copied to clipboard
| Challenge: | End-to-end speech translation models learn acoustic representations from the encoder, which is not desirable for cross-modal and cross-lingual translation. |
| Approach: | They propose an adaptive speech-to-text translation model that dynamically adapts acoustic states in the decoder. |
| Outcome: | The proposed model outperforms state-of-the-art speech translation models on two widely-used datasets. |
InfiniSST: Simultaneous Translation of Unbounded Speech with Large Language Model (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing models for simultaneous speech translation assume pre-segmented speech, limiting their real-world applicability. |
| Approach: | They propose a multi-turn dialogue task that can translate unbounded streaming speech . they construct translation trajectories and robust segments from MuST-C with multi-latency augmentation during training and develop a cache management strategy to facilitate efficient inference. |
| Outcome: | The proposed approach reduces computation-aware latency by 0.5 to 1 second while maintaining the same translation quality compared to baselines. |