Efficient CTC Regularization via Coarse Labels for End-to-End Speech Translation (2023.eacl-main)
Copied to clipboard
| Challenge: | Developing techniques to support end-to-end speech translation is non-trivial because of the speech-text modality gap. |
| Approach: | They propose a coarse labeling approach that merges vocabulary labels via simple heuristic rules . they propose to use 256-bit truncation, division or modulo operations to regularize the encoder . |
| Outcome: | The proposed method can increase training efficiency while delivering better performance. |
Similar Papers
Investigating the Reordering Capability in CTC-based Non-Autoregressive End-to-End Speech Translation (2021.findings-acl)
Copied to clipboard
| Challenge: | Using connectionist temporal classification (CTC) for speech-to-text translation is counter-intuitive due to its monotonicity assumption. |
| Approach: | They propose to build a non-autoregressive speech-to-text translation model using connectionist temporal classification (CTC) their work shows transformer encoders can change the word order and points out the future research direction that needs to be explored more on non-Autoregressives speech translation. |
| Outcome: | The proposed model improves translation performance by using transformer encoders. |
CTC-based Compression for Direct Speech Translation (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing studies have shown that a dynamic phone-informed compression of the input audio is beneficial for speech translation (ST). |
| Approach: | They propose a method which performs a phone-informed compression of the input audio in direct ST models by exploiting the Connectionist Temporal Classification (CTC) they demonstrate that their method brings a 1.3-1.5 BLEU improvement over a strong baseline on two language pairs (English-Italian and English-German) |
| Outcome: | The proposed method brings a 1.3-1.5 BLEU improvement over a strong baseline on two language pairs (English-Italian and English-German) it reduces memory footprint by more than 10%, and is faster than previous approaches. |
BERT Meets CTC: New Formulation of End-to-End Speech Recognition with Pre-trained Masked Language Model (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to connectionist temporal classification (CTC) are based on pre-trained language models (LMs) |
| Approach: | They propose a formulation of connectionist temporal classification that relaxes the conditional independence assumptions used in conventional CTC and incorporates linguistic knowledge through explicit output dependency. |
| Outcome: | The proposed model improves over conventional approaches across variations in speaking styles and languages while maintaining CTC’s training efficiency. |
CTC-based Non-autoregressive Speech Translation (2023.acl-long)
Copied to clipboard
Chen Xu, Xiaoqian Liu, Xiaowen Liu, Qingxuan Sun, Yuhao Zhang, Murun Yang, Qianqian Dong, Tom Ko, Mingxuan Wang, Tong Xiao, Anxiang Ma, Jingbo Zhu
| Challenge: | End-to-end speech translation (E2E ST) and non-autoregressive (NAR) generation are promising in language and speech processing for their advantages of less error propagation and low latency. |
| Approach: | They develop a model that uses connectionist temporal classification to predict the source and target texts. |
| Outcome: | The proposed model achieves an average BLEU score of 29.5 with a speed-up of 5.67. |
Exploring Phoneme-Level Speech Representations for End-to-End Speech Translation (P19-1)
Copied to clipboard
| Challenge: | Previous work on end-to-end translation from speech uses frame-level features as speech representations, which creates longer, sparser sequences than text. |
| Approach: | They propose a method to generate compressed phoneme-like speech representations that generate shorter, higher-level source sequences for translation. |
| Outcome: | The proposed method improves translation performance by 5 BLEU on high and low resource languages and reduces training time by 60%. |
CTC Alignments Improve Autoregressive Translation (2023.eacl-main)
Copied to clipboard
Brian Yan, Siddharth Dalmia, Yosuke Higuchi, Graham Neubig, Florian Metze, Alan W Black, Shinji Watanabe
| Challenge: | Connectionist Temporal Classification (CTC) is widely used for automatic speech recognition (ASR) but lags behind attentional decoder approaches in terms of translation quality. |
| Approach: | They propose to use a CTC/attention framework to validate this hypothesis by modifying the Hybrid CTC-Attention model proposed for automatic speech recognition to support text-to-text translation (MT) and speech-totext translation. |
| Outcome: | The proposed model outperforms pure-attention baselines across six translation tasks. |
Simple and Effective Unsupervised Speech Translation (2023.acl-long)
Copied to clipboard
Changhan Wang, Hirofumi Inaguma, Peng-Jen Chen, Ilia Kulikov, Yun Tang, Wei-Ning Hsu, Michael Auli, Juan Pino
| Challenge: | Existing methods to train speech models without labeled data are limited for most languages. |
| Approach: | They propose a pipeline approach to build speech translation systems without labeled data by leveraging recent advances in unsupervised speech recognition, machine translation and speech synthesis. |
| Outcome: | The proposed approach outperforms the state-of-the-art in unsupervised speech recognition by 3.2 BLEU on the Libri-Trans benchmark and the best supervised end-to-end models from only two years ago by an average of 5.0 BLUE over five X-En directions. |
Bringing Emerging Architectures to Sequence Labeling in NLP (2026.eacl-long)
Copied to clipboard
| Challenge: | Pretrained Transformer encoders are the dominant approach to sequence labeling . however, few have been applied to sequence labels on flat or simplified tasks . |
| Approach: | They propose to use pretrained Transformer encoders to model relations across words . they find that the architectures adapt well across tagging tasks that vary in complexity . |
| Outcome: | The proposed architectures perform well across tagging tasks across languages and datasets. |
Understanding and Bridging the Modality Gap for Speech Translation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods to improve end-to-end speech translation (ST) use multitask learning, but there is always a modality gap between ST and MT due to the differences between speech and text. |
| Approach: | They propose a method to bridge the modality gap between ST and MT by leveraging (text) machine translation data. |
| Outcome: | The proposed method bridges the modality gap and achieves significant improvements over baseline in all eight directions. |
An Empirical Study of Consistency Regularization for End-to-End Speech-to-Text Translation (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for speech-to-text translation (ST) have achieved impressive supervised and zero-shot performance. |
| Approach: | They propose to use consistency regularization methods to boost end-to-end (E2E) speech-totext translation (ST) by regularizing the intra-modal consistency instead of the modality gap. |
| Outcome: | The proposed training strategies achieve state-of-the-art (SOTA) performance in most translation directions. |