Chen Xu, Xiaoqian Liu, Xiaowen Liu, Qingxuan Sun, Yuhao Zhang, Murun Yang, Qianqian Dong, Tom Ko, Mingxuan Wang, Tong Xiao, Anxiang Ma, Jingbo Zhu
| Challenge: | End-to-end speech translation (E2E ST) and non-autoregressive (NAR) generation are promising in language and speech processing for their advantages of less error propagation and low latency. |
| Approach: | They develop a model that uses connectionist temporal classification to predict the source and target texts. |
| Outcome: | The proposed model achieves an average BLEU score of 29.5 with a speed-up of 5.67. |
Similar Papers
Investigating the Reordering Capability in CTC-based Non-Autoregressive End-to-End Speech Translation (2021.findings-acl)
Copied to clipboard
| Challenge: | Using connectionist temporal classification (CTC) for speech-to-text translation is counter-intuitive due to its monotonicity assumption. |
| Approach: | They propose to build a non-autoregressive speech-to-text translation model using connectionist temporal classification (CTC) their work shows transformer encoders can change the word order and points out the future research direction that needs to be explored more on non-Autoregressives speech translation. |
| Outcome: | The proposed model improves translation performance by using transformer encoders. |
CTC Alignments Improve Autoregressive Translation (2023.eacl-main)
Copied to clipboard
Brian Yan, Siddharth Dalmia, Yosuke Higuchi, Graham Neubig, Florian Metze, Alan W Black, Shinji Watanabe
| Challenge: | Connectionist Temporal Classification (CTC) is widely used for automatic speech recognition (ASR) but lags behind attentional decoder approaches in terms of translation quality. |
| Approach: | They propose to use a CTC/attention framework to validate this hypothesis by modifying the Hybrid CTC-Attention model proposed for automatic speech recognition to support text-to-text translation (MT) and speech-totext translation. |
| Outcome: | The proposed model outperforms pure-attention baselines across six translation tasks. |
End-to-End Non-Autoregressive Neural Machine Translation with Connectionist Temporal Classification (D18-1)
Copied to clipboard
| Challenge: | Autoregressive decoding is the only part of sequence-to-sequence models that prevents massive parallelization at inference time. |
| Approach: | They propose a non-autoregressive architecture based on connectionist temporal classification . they conduct experiments on the WMT English-Romanian and English-German datasets . |
| Outcome: | The proposed model achieves a significant speedup over autoregressive models . the model can be trained end-to-end and maintains translation quality comparable to other models compared to autoregression models based on connectionist temporal classification . |
CTC-based Non-autoregressive Textless Speech-to-Speech Translation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing direct speech-to-speech translation models require text supervision during training, which is not feasible for numerous unwritten languages. |
| Approach: | They propose a non-autoregressive (NAR) model that generates discrete units from the source speech and employs a unit-based vocoder to synthesize the target. |
| Outcome: | The proposed model achieves translation quality comparable to the autoregressive model while preserving up to 26.81 decoding speedup. |
A Non-autoregressive Generation Framework for End-to-End Simultaneous Speech-to-Any Translation (2024.acl-long)
Copied to clipboard
| Challenge: | Existing translation pipelines require additional cascade components to achieve speech-to-speech translation. |
| Approach: | They propose a non-autoregressive generation framework for simultaneous speech translation . it integrates both text-to-text and speech-tospeech tasks into a unified framework . |
| Outcome: | The proposed framework outperforms state-of-the-art models in speech-to-text and speech- to-speech tasks. |
Context-Aware Non-Autoregressive Document-Level Translation with Sentence-Aligned Connectionist Temporal Classification (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies employ autoregressive translation (AT) methods to encode sentences . however, the AT methods struggle with error accumulation when the length of sentences increases. |
| Approach: | They propose a context-aware non-autoregressive framework with the sentence-aligned connectionist temporal classification loss for document-level neural machine translation. |
| Outcome: | The proposed framework achieves 46X speedup on three benchmarks compared to strong baselines. |
Efficient CTC Regularization via Coarse Labels for End-to-End Speech Translation (2023.eacl-main)
Copied to clipboard
| Challenge: | Developing techniques to support end-to-end speech translation is non-trivial because of the speech-text modality gap. |
| Approach: | They propose a coarse labeling approach that merges vocabulary labels via simple heuristic rules . they propose to use 256-bit truncation, division or modulo operations to regularize the encoder . |
| Outcome: | The proposed method can increase training efficiency while delivering better performance. |
Non-autoregressive Streaming Transformer for Simultaneous Translation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Simultaneous machine translation models are trained to strike a balance between latency and translation quality. |
| Approach: | They propose a non-autoregressive streaming Transformer which generates blank tokens and decodes repetitive tokens to adjust its READ/WRITE strategy flexibly. |
| Outcome: | The proposed model outperforms previous strong autoregressive models on various benchmarks on siMT. |
CTC-based Compression for Direct Speech Translation (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing studies have shown that a dynamic phone-informed compression of the input audio is beneficial for speech translation (ST). |
| Approach: | They propose a method which performs a phone-informed compression of the input audio in direct ST models by exploiting the Connectionist Temporal Classification (CTC) they demonstrate that their method brings a 1.3-1.5 BLEU improvement over a strong baseline on two language pairs (English-Italian and English-German) |
| Outcome: | The proposed method brings a 1.3-1.5 BLEU improvement over a strong baseline on two language pairs (English-Italian and English-German) it reduces memory footprint by more than 10%, and is faster than previous approaches. |
Non-Autoregressive Neural Machine Translation: A Call for Clarity (2022.emnlp-main)
Copied to clipboard
| Challenge: | Non-autoregressive translation models require a single forward pass to generate the output sequence instead of iteratively producing each predicted token. |
| Approach: | They propose to use a single forward pass to generate the output sequence instead of iteratively producing each predicted token. |
| Outcome: | The proposed models improve translation quality and speed under third-party testing environments. |