Textless Speech-to-Speech Translation With Limited Parallel Data (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing speech-to-speech translation models either leverage text as an intermediate step or require hundreds of hours of parallel speech data. |
| Approach: | They propose a framework for training textless S2ST models that require dozens of hours of parallel speech data. |
| Outcome: | The proposed model achieves reasonable performance on three domains with single-speaker synthesized speech. |
Similar Papers
Textless Speech-to-Speech Translation on Real Data (2022.naacl-main)
Copied to clipboard
Ann Lee, Hongyu Gong, Paul-Ambroise Duquenne, Holger Schwenk, Peng-Jen Chen, Changhan Wang, Sravya Popuri, Yossi Adi, Juan Pino, Jiatao Gu, Wei-Ning Hsu
| Challenge: | Existing text-based speech-to-speech translation systems rely on cascaded approach . text-to text translation systems require text generation and a single input to generate output . |
| Approach: | They propose a textless speech-to-speech translation system that can translate speech from one language into another without the need of text data. |
| Outcome: | The proposed system can translate speech from one language into another without text data. |
Can We Achieve High-quality Direct Speech-to-Speech Translation without Parallel Speech Data? (2024.acl-long)
Copied to clipboard
| Challenge: | Existing two-pass direct speech-to-speech translation models require parallel speech data to train, which is challenging to collect. |
| Approach: | They propose a two-pass direct speech-to-speech translation (S2ST) model that decomposes the task into speech- to-text translation (s2TT) and text-tospech (TTS) they propose 'composer' S2ST model that integrates pretrained S2TT and TTS models into a direct S2 ST model. |
| Outcome: | The proposed model integrates pretrained S2TT and TTS models into a direct S2ST model without parallel speech data. |
Direct Speech-to-Speech Translation With Discrete Units (2022.acl-long)
Copied to clipboard
Ann Lee, Peng-Jen Chen, Changhan Wang, Jiatao Gu, Sravya Popuri, Xutai Ma, Adam Polyak, Yossi Adi, Qing He, Yun Tang, Juan Pino, Wei-Ning Hsu
| Challenge: | Existing direct speech-to-speech translation models rely on text generation as an intermediate step. |
| Approach: | They propose a direct speech-to-speech translation model that translates speech from one language to another without relying on intermediate text generation. |
| Outcome: | The proposed model produces 6.7 BLEUs in the Fisher Spanish-English dataset when trained without any text transcripts and with text supervision. |
Speech-to-Speech Translation for a Real-world Unwritten Language (2023.findings-acl)
Copied to clipboard
Peng-Jen Chen, Kevin Tran, Yilin Yang, Jingfei Du, Justine Kao, Yu-An Chung, Paden Tomasello, Paul-Ambroise Duquenne, Holger Schwenk, Hongyu Gong, Hirofumi Inaguma, Sravya Popuri, Changhan Wang, Juan Pino, Wei-Ning Hsu, Ann Lee
| Challenge: | a new study examines speech-to-speech translation (S2ST) that translates speech from one language into another . the research area for unwritten languages remains a research area with little exploration due to the lack of training data. |
| Approach: | They propose a system that translates speech from one language into another . they use Taiwanese Hokkien as an example of an unwritten language . |
| Outcome: | The proposed system can be used to train models in languages without standard writing systems. |
Dub-S2ST: Textless Speech-to-Speech Translation for Seamless Dubbing (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing speech translation approaches often overlook the transfer of speech patterns, leading to mismatches with source speech and limiting their suitability for dubbing applications. |
| Approach: | They propose a diffusion-based speech-to-unit translation model with explicit duration control that enables time-aligned translation. |
| Outcome: | The proposed system preserves key characteristics such as duration, speaker identity, and speaking speed while maintaining key characteristics. |
Speech-to-Speech Translation with Discrete-Unit-Based Style Transfer (2024.acl-srw)
Copied to clipboard
| Challenge: | Existing methods to translate spoken utterances from one language to another are unable to preserve speaker timbre of source speech. |
| Approach: | They propose a pipeline with style-transfer capability on the basis of self-supervised speech representations and codec units. |
| Outcome: | The proposed model achieves zero-shot cross-lingual style transfer on previously unseen source languages. |
Back Translation for Speech-to-text Translation Without Transcripts (2023.acl-long)
Copied to clipboard
| Challenge: | End-to-end speech-totext translation (ST) is often achieved by utilizing source transcripts, but transcripts are only sometimes available since numerous unwritten languages exist worldwide. |
| Approach: | They propose an algorithm to synthesize pseudo ST data from monolingual target data to enhance ST without generating source transcripts. |
| Outcome: | The proposed method achieves an average boost of 2.3 BLEU on MuST-C En-De, En-Fr, and En-Es datasets. |
Leveraging Unit Language Guidance to Advance Speech Modeling in Textless Speech-to-Speech Translation (2025.findings-acl)
Copied to clipboard
Yuhao Zhang, Xiangnan Ma, Kaiqi Kou, Peizhuo Liu, Weiqiao Shan, Benyou Wang, Tong Xiao, Yuxin Huang, Zhengtao Yu, JingBo Zhu
| Challenge: | Existing textless speech-to-speech translation models have two main challenges: 1) learning cross-modal features and 2) learning alignment of difference languages in long sequences. |
| Approach: | They propose a unit language to overcome two main modeling challenges . they propose task prompt modeling to utilize the unit language in guiding the modeling process. |
| Outcome: | The proposed language improves over a strong baseline and achieves comparable performance to models trained with text. |
From Tens of Hours to Tens of Thousands: Scaling Back-Translation for Speech Recognition (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in Automatic Speech Recognition (ASR) have been fueled by massive speech corpora, but extending coverage to diverse languages with limited resources remains a formidable challenge. |
| Approach: | They propose a pipeline that converts large-scale text corpora into synthetic speech using off-the-shelf text-to-speech (TTS) models. |
| Outcome: | The proposed pipeline generates 500,000 hours of synthetic speech in ten languages and achieves transcription error reductions of over 30%. |
LibriS2S: A German-English Speech-to-Speech Translation Corpus (2022.lrec-1)
Copied to clipboard
| Challenge: | Recent advances in speech-to-text translation have led to significant improvements, but the availability of appropriate training data is limiting. |
| Approach: | They propose a new text-to-speech and speech-tospech translation model that directly learns to generate the speech signal based on the pronunciation of the source language. |
| Outcome: | The proposed model learns to generate speech signal based on pronunciation of source language. |