InfiniSST: Simultaneous Translation of Unbounded Speech with Large Language Model (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing models for simultaneous speech translation assume pre-segmented speech, limiting their real-world applicability. |
| Approach: | They propose a multi-turn dialogue task that can translate unbounded streaming speech . they construct translation trajectories and robust segments from MuST-C with multi-latency augmentation during training and develop a cache management strategy to facilitate efficient inference. |
| Outcome: | The proposed approach reduces computation-aware latency by 0.5 to 1 second while maintaining the same translation quality compared to baselines. |
Similar Papers
StreamAtt: Direct Streaming Speech-to-Text Translation with Attention-based Audio History Selection (2024.acl-long)
Copied to clipboard
| Challenge: | Existing studies on streaming translation focus on SimulST only focusing on StreamST . StreamAtt is the first Stream ST policy and proposes StreamLAAL . |
| Approach: | They propose StreamAtt, the first StreamST policy, and StreamLAAL, the second Stream ST latency metric. |
| Outcome: | Experiments in 8 languages show that StreamAtt is more efficient than SimulST . StreamLAAL is the first StreamST latency metric comparable with existing metrics for Simul ST. |
Simul-MuST-C: Simultaneous Multilingual Speech Translation Corpus Using Large Language Model (2024.emnlp-main)
Copied to clipboard
| Challenge: | Simultaneous speech translation (SiST) begins translating before the entire source input is received. |
| Approach: | They propose a dataset that rearranges sentences into segmented monotonic data for simultaneous speech translation using the Large Language Model. |
| Outcome: | The proposed dataset improves quality and latency in siST translations by rearranging sentences into segmented monotonic data. |
Fluent and Low-latency Simultaneous Speech-to-Speech Translation with Self-adaptive Training (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Current approaches to simultaneous speech-to-speech translation accumulate more and more latencies in later sentences when the speaker talks faster. |
| Approach: | They propose a method which generates more fluent target speech latency than the baseline . they propose to use self-adaptive translation to adjust the length of translations to accommodate different source speech rates. |
| Outcome: | Xiong et al., 2019) show that the proposed method generates more fluent target speech latency than baseline . authors say it provides more natural communication process than speech-to-text translation . xiong and colleagues say the proposed technique is more efficient than current approaches . |
Hierarchical Policy Optimization for Simultaneous Translation of Unbounded Speech (2026.acl-long)
Copied to clipboard
| Challenge: | Existing synthesis methods cannot guarantee data quality. |
| Approach: | They propose a hierarchical reward that balances translation quality and latency objectives by combining supervised fine-tuning data with supervised inputs. |
| Outcome: | The proposed model can reuse key-value caches across both modalities and eliminate redundant feature recomputation. |
Does Simultaneous Speech Translation need Simultaneous Models? (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Simultaneous speech translation (SimulST) systems strive for high output quality but also low latency. |
| Approach: | They propose to train SimulST offline without additional training or adaptation . they also show offline training achieves similar or better quality compared to offline training . |
| Outcome: | The proposed model can serve both offline and simultaneous applications without additional training or adaptation. |
Learning When to Translate for Streaming Speech (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods waiting-and-translating for a fixed duration break speech acoustic units . Existing models waiting-for a set duration and generating partial sentences are not effective . |
| Approach: | They propose a monotonic segmentation module inside an encoder-decoder model to detect proper speech unit boundaries for a streaming speech input. |
| Outcome: | The proposed method outperforms existing methods on a speech translation dataset and achieves the best trade-off between translation quality and latency. |
Direct Simultaneous Speech-to-Text Translation Assisted by Synchronized Streaming ASR (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to simultaneous speech-to-text translation suffer from error propagation and extra latency. |
| Approach: | They propose a new paradigm for simultaneous speech-to-text translation using two separate decoders . they use multitask learning to jointly learn these two tasks with a shared encoder . |
| Outcome: | The proposed method achieves substantially better translation quality at similar levels of latency. |
SimulS2S-LLM: Unlocking Simultaneous Inference of Speech LLMs for Speech-to-Speech Translation (2025.acl-long)
Copied to clipboard
| Challenge: | Simultaneous speech translation (SST) outputs translations in parallel with streaming speech input, balancing translation quality and latency. |
| Approach: | They propose to train LLMs offline and employ a test-time policy to guide simultaneous inference by extracting boundary-aware speech prompts that allow it to be better matched with text input data. |
| Outcome: | The proposed model trains speech LLMs offline and employs a test-time policy to guide simultaneous inference. |
CA*: Addressing Evaluation Pitfalls in Computation-Aware Latency for Simultaneous Speech Translation (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing metrics for Simultaneous speech translation (SimulST) are inaccurately measuring latency in unsegmented streaming settings. |
| Approach: | They propose to modify existing metrics to correctly measure computation-aware latency for SimulST systems, addressing limitations present in existing metrics. |
| Outcome: | The proposed model is based on a real-time, lowlatency scenario where the model starts generating the textual translation before the entire audio input is processed. |
RealTranS: End-to-End Simultaneous Speech Translation with Convolutional Weighted-Shrinking Transformer (2021.findings-acl)
Copied to clipboard
| Challenge: | End-to-end simultaneous speech translation (SST) is useful in many scenarios but has not been fully investigated. |
| Approach: | They propose an end-to-end simultaneous speech translation model called RealTranS . they use interleaved convolution and unidirectional Transformer layers to downsample input speech . |
| Outcome: | The proposed model outperforms existing models on public and widely-used datasets in multiple latency settings. |