StreamSpeech: Simultaneous Speech-to-Speech Translation with Multi-task Learning (2024.acl-long)
Copied to clipboard
| Challenge: | Existing simultaneous translation methods focus on text-to-text and speech-totext translation. |
| Approach: | They propose a Simul-S2ST model that jointly learns translation and simultaneous policy in a unified framework of multi-task learning. |
| Outcome: | The proposed model can perform offline and simultaneous speech recognition, speech translation and speech synthesis via an "All-in-One" seamless model. |
Similar Papers
SimulSpeech: End-to-End Simultaneous Speech to Text Translation (2020.acl-main)
Copied to clipboard
| Challenge: | SimulSpeech is an end-to-end simultaneous speech to text translation system . conventional approaches to simultaneous speech translation divide the translation process into two stages . |
| Approach: | They develop an end-to-end simultaneous speech to text translation system which translates speech in source language to text in target language concurrently. |
| Outcome: | The proposed system achieves reasonable BLEU scores and lower delay compared to full-sentence translation model. |
Direct Simultaneous Speech-to-Text Translation Assisted by Synchronized Streaming ASR (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to simultaneous speech-to-text translation suffer from error propagation and extra latency. |
| Approach: | They propose a new paradigm for simultaneous speech-to-text translation using two separate decoders . they use multitask learning to jointly learn these two tasks with a shared encoder . |
| Outcome: | The proposed method achieves substantially better translation quality at similar levels of latency. |
SimulS2S-LLM: Unlocking Simultaneous Inference of Speech LLMs for Speech-to-Speech Translation (2025.acl-long)
Copied to clipboard
| Challenge: | Simultaneous speech translation (SST) outputs translations in parallel with streaming speech input, balancing translation quality and latency. |
| Approach: | They propose to train LLMs offline and employ a test-time policy to guide simultaneous inference by extracting boundary-aware speech prompts that allow it to be better matched with text input data. |
| Outcome: | The proposed model trains speech LLMs offline and employs a test-time policy to guide simultaneous inference. |
AV-TranSpeech: Audio-Visual Robust Speech-to-Speech Translation (2023.acl-long)
Copied to clipboard
Rongjie Huang, Huadai Liu, Xize Cheng, Yi Ren, Linjun Li, Zhenhui Ye, Jinzheng He, Lichao Zhang, Jinglin Liu, Xiang Yin, Zhou Zhao
| Challenge: | Existing models for speech-to-speech translation suffer from distinct degradation in noisy environments and fail to translate visual speech. |
| Approach: | They propose a text-based audio-visual speech-to-speech translation model that integrates visual information with audio-only data to improve system robustness. |
| Outcome: | The proposed model outperforms models trained on audio-only corpus in two languages . it also improves with low-resource audio-visual data, compared with baselines . |
Can We Achieve High-quality Direct Speech-to-Speech Translation without Parallel Speech Data? (2024.acl-long)
Copied to clipboard
| Challenge: | Existing two-pass direct speech-to-speech translation models require parallel speech data to train, which is challenging to collect. |
| Approach: | They propose a two-pass direct speech-to-speech translation (S2ST) model that decomposes the task into speech- to-text translation (s2TT) and text-tospech (TTS) they propose 'composer' S2ST model that integrates pretrained S2TT and TTS models into a direct S2 ST model. |
| Outcome: | The proposed model integrates pretrained S2TT and TTS models into a direct S2ST model without parallel speech data. |
StreamAtt: Direct Streaming Speech-to-Text Translation with Attention-based Audio History Selection (2024.acl-long)
Copied to clipboard
| Challenge: | Existing studies on streaming translation focus on SimulST only focusing on StreamST . StreamAtt is the first Stream ST policy and proposes StreamLAAL . |
| Approach: | They propose StreamAtt, the first StreamST policy, and StreamLAAL, the second Stream ST latency metric. |
| Outcome: | Experiments in 8 languages show that StreamAtt is more efficient than SimulST . StreamLAAL is the first StreamST latency metric comparable with existing metrics for Simul ST. |
InfiniSST: Simultaneous Translation of Unbounded Speech with Large Language Model (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing models for simultaneous speech translation assume pre-segmented speech, limiting their real-world applicability. |
| Approach: | They propose a multi-turn dialogue task that can translate unbounded streaming speech . they construct translation trajectories and robust segments from MuST-C with multi-latency augmentation during training and develop a cache management strategy to facilitate efficient inference. |
| Outcome: | The proposed approach reduces computation-aware latency by 0.5 to 1 second while maintaining the same translation quality compared to baselines. |
Streaming Models for Joint Speech Recognition and Translation (2021.eacl-main)
Copied to clipboard
| Challenge: | Using end-to-end models for speech translation has become a focus of the ST community . cascaded models have the advantage of including automatic speech recognition output . |
| Approach: | They propose a model that condenses sound waves into translated text and integrates automatic speech recognition outputs into the models. |
| Outcome: | The proposed model is statistically similar to cascading models, but has half the number of parameters. |
Does Simultaneous Speech Translation need Simultaneous Models? (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Simultaneous speech translation (SimulST) systems strive for high output quality but also low latency. |
| Approach: | They propose to train SimulST offline without additional training or adaptation . they also show offline training achieves similar or better quality compared to offline training . |
| Outcome: | The proposed model can serve both offline and simultaneous applications without additional training or adaptation. |
Adapting Offline Speech Translation Models for Streaming with Future-Aware Distillation and Inference (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to streaming speech translation use an offline model with a wait-k policy . however, there is a mismatch problem with an offline inference model trained with complete utterances . |
| Approach: | They propose an offline streaming speech translation model with wait-k policy to support different latency requirements. |
| Outcome: | The proposed model achieves better trade-offs between translation quality and latency than baselines. |