Leveraging Unit Language Guidance to Advance Speech Modeling in Textless Speech-to-Speech Translation (2025.findings-acl)
Copied to clipboard
Yuhao Zhang, Xiangnan Ma, Kaiqi Kou, Peizhuo Liu, Weiqiao Shan, Benyou Wang, Tong Xiao, Yuxin Huang, Zhengtao Yu, JingBo Zhu
| Challenge: | Existing textless speech-to-speech translation models have two main challenges: 1) learning cross-modal features and 2) learning alignment of difference languages in long sequences. |
| Approach: | They propose a unit language to overcome two main modeling challenges . they propose task prompt modeling to utilize the unit language in guiding the modeling process. |
| Outcome: | The proposed language improves over a strong baseline and achieves comparable performance to models trained with text. |
Similar Papers
Direct Speech-to-Speech Translation With Discrete Units (2022.acl-long)
Copied to clipboard
Ann Lee, Peng-Jen Chen, Changhan Wang, Jiatao Gu, Sravya Popuri, Xutai Ma, Adam Polyak, Yossi Adi, Qing He, Yun Tang, Juan Pino, Wei-Ning Hsu
| Challenge: | Existing direct speech-to-speech translation models rely on text generation as an intermediate step. |
| Approach: | They propose a direct speech-to-speech translation model that translates speech from one language to another without relying on intermediate text generation. |
| Outcome: | The proposed model produces 6.7 BLEUs in the Fisher Spanish-English dataset when trained without any text transcripts and with text supervision. |
Textless Speech-to-Speech Translation on Real Data (2022.naacl-main)
Copied to clipboard
Ann Lee, Hongyu Gong, Paul-Ambroise Duquenne, Holger Schwenk, Peng-Jen Chen, Changhan Wang, Sravya Popuri, Yossi Adi, Juan Pino, Jiatao Gu, Wei-Ning Hsu
| Challenge: | Existing text-based speech-to-speech translation systems rely on cascaded approach . text-to text translation systems require text generation and a single input to generate output . |
| Approach: | They propose a textless speech-to-speech translation system that can translate speech from one language into another without the need of text data. |
| Outcome: | The proposed system can translate speech from one language into another without text data. |
Dub-S2ST: Textless Speech-to-Speech Translation for Seamless Dubbing (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing speech translation approaches often overlook the transfer of speech patterns, leading to mismatches with source speech and limiting their suitability for dubbing applications. |
| Approach: | They propose a diffusion-based speech-to-unit translation model with explicit duration control that enables time-aligned translation. |
| Outcome: | The proposed system preserves key characteristics such as duration, speaker identity, and speaking speed while maintaining key characteristics. |
Speech-to-Speech Translation for a Real-world Unwritten Language (2023.findings-acl)
Copied to clipboard
Peng-Jen Chen, Kevin Tran, Yilin Yang, Jingfei Du, Justine Kao, Yu-An Chung, Paden Tomasello, Paul-Ambroise Duquenne, Holger Schwenk, Hongyu Gong, Hirofumi Inaguma, Sravya Popuri, Changhan Wang, Juan Pino, Wei-Ning Hsu, Ann Lee
| Challenge: | a new study examines speech-to-speech translation (S2ST) that translates speech from one language into another . the research area for unwritten languages remains a research area with little exploration due to the lack of training data. |
| Approach: | They propose a system that translates speech from one language into another . they use Taiwanese Hokkien as an example of an unwritten language . |
| Outcome: | The proposed system can be used to train models in languages without standard writing systems. |
Can We Achieve High-quality Direct Speech-to-Speech Translation without Parallel Speech Data? (2024.acl-long)
Copied to clipboard
| Challenge: | Existing two-pass direct speech-to-speech translation models require parallel speech data to train, which is challenging to collect. |
| Approach: | They propose a two-pass direct speech-to-speech translation (S2ST) model that decomposes the task into speech- to-text translation (s2TT) and text-tospech (TTS) they propose 'composer' S2ST model that integrates pretrained S2TT and TTS models into a direct S2 ST model. |
| Outcome: | The proposed model integrates pretrained S2TT and TTS models into a direct S2ST model without parallel speech data. |
Toward Joint Language Modeling for Speech Units and Text (2023.findings-emnlp)
Copied to clipboard
Ju-Chieh Chou, Chung-Ming Chien, Wei-Ning Hsu, Karen Livescu, Arun Babu, Alexis Conneau, Alexei Baevski, Michael Auli
| Challenge: | Speech and text are two major forms of human language and little effort has been made to model them together. |
| Approach: | They propose to combine speech and text models to create mixed speech-text data by using different tokenizers and automatic metrics to evaluate how well the model mixes speech and texts. |
| Outcome: | The proposed model improves over a speech-only baseline and shows zero-shot cross-modal transferability. |
Speech-to-Speech Translation with Discrete-Unit-Based Style Transfer (2024.acl-srw)
Copied to clipboard
| Challenge: | Existing methods to translate spoken utterances from one language to another are unable to preserve speaker timbre of source speech. |
| Approach: | They propose a pipeline with style-transfer capability on the basis of self-supervised speech representations and codec units. |
| Outcome: | The proposed model achieves zero-shot cross-lingual style transfer on previously unseen source languages. |
PromptST: Abstract Prompt Learning for End-to-End Speech Translation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that PromptST can improve speech-to-text translation by capturing richer linguistic knowledge. |
| Approach: | They propose a plug-in prompt-enhanced S2T model that captures richer linguistic knowledge . they use a 10GB linguistic probing benchmark to investigate the fusion of speech and text features . |
| Outcome: | The proposed model can improve on a strong baseline by capturing richer linguistic knowledge. |
Back Translation for Speech-to-text Translation Without Transcripts (2023.acl-long)
Copied to clipboard
| Challenge: | End-to-end speech-totext translation (ST) is often achieved by utilizing source transcripts, but transcripts are only sometimes available since numerous unwritten languages exist worldwide. |
| Approach: | They propose an algorithm to synthesize pseudo ST data from monolingual target data to enhance ST without generating source transcripts. |
| Outcome: | The proposed method achieves an average boost of 2.3 BLEU on MuST-C En-De, En-Fr, and En-Es datasets. |
Leveraging Large Pre-trained Multilingual Models for High-Quality Speech-to-Text Translation on Industry Scenarios (2025.coling-main)
Copied to clipboard
| Challenge: | Speech-to-Text Translation systems rely on a sequential pipeline that combines ASR and MT models. |
| Approach: | They propose a parameter-efficient framework that integrates one LPSM with a multilingual MT engine. |
| Outcome: | The proposed framework integrates one LPSM with a multilingual MT engine. |