Hybrid Transducer and Attention based Encoder-Decoder Modeling for Speech-to-Text Tasks (2023.acl-long)
Copied to clipboard
| Challenge: | Neural based end-to-end frameworks have achieved remarkable success in speech-totext tasks, such as automatic speech recognition (ASR) and speech- totext translation (ST). |
| Approach: | They propose to combine Transducer and Attention based Encoder-Decoder (TAED) for speech-to-text tasks and leverage AED's strength in non-monotonic sequence to sequence learning while retaining Transducers streaming property. |
| Outcome: | The proposed model outperforms Transducer and Attention based Encoder-Decoder (TAED) on the MuST-C dataset and shows that it is not bound by any specific language model. |
Similar Papers
AdaST: Dynamically Adapting Encoder States in the Decoder for End-to-End Speech-to-Text Translation (2021.findings-acl)
Copied to clipboard
| Challenge: | End-to-end speech translation models learn acoustic representations from the encoder, which is not desirable for cross-modal and cross-lingual translation. |
| Approach: | They propose an adaptive speech-to-text translation model that dynamically adapts acoustic states in the decoder. |
| Outcome: | The proposed model outperforms state-of-the-art speech translation models on two widely-used datasets. |
Cross Attention Augmented Transducer Networks for Simultaneous Translation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to simultaneous translation are limited by monotonic constraint . a novel architecture for simultaneous translation is proposed . |
| Approach: | They propose a cross attention-augmented transducer for simultaneous translation that optimizes both policies and translation models by expanding target sequences with blank symbols. |
| Outcome: | The proposed architecture achieves better latency-quality trade-offs than state-of-the-art approaches. |
Unified Speech-Text Pre-training for Speech Translation and Recognition (2022.acl-long)
Copied to clipboard
Yun Tang, Hongyu Gong, Ning Dong, Changhan Wang, Wei-Ning Hsu, Jiatao Gu, Alexei Baevski, Xian Li, Abdelrahman Mohamed, Michael Auli, Juan Pino
| Challenge: | Existing methods to pre-train speech and text use unlabeled data to learn universal feature representations. |
| Approach: | They propose a method to jointly pre-train speech and text in an encoder-decoder modeling framework for speech translation and recognition. |
| Outcome: | The proposed method achieves between 1.7 and 2.3 BLEU improvement above the state of the art on the MuST-C speech translation dataset and comparable WERs to wav2vec 2.0 on the Librispeech speech recognition task. |
SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing (2022.acl-long)
Copied to clipboard
Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, Zhihua Wei, Yao Qian, Jinyu Li, Furu Wei
| Challenge: | Existing work shows that pre-trained models can improve in various natural language processing tasks. |
| Approach: | They propose a unified-modal encoder-decoder framework that pre-trains speech-text representations using large-scale unlabeled speech and text data. |
| Outcome: | The proposed framework is superior to existing models on speech-to-text processing tasks. |
MultiMed: Multilingual Medical Speech Recognition via Attention Encoder Decoder (2025.acl-industry)
Copied to clipboard
Khai Le-Duc, Phuc Phan, Tan-Hanh Pham, Bach Phan Tat, Minh-Huong Ngo, Thanh Nguyen-Tang, Truong-Son Hy
| Challenge: | Multilingual automatic speech recognition (ASR) in the medical domain is a critical foundational task, serving a wide range of downstream applications such as speech translation, spoken language understanding, and voice-activated assistants. |
| Approach: | They present the first multilingual medical ASR dataset and the first collection of small-to-large end-to end medical APR models spanning five languages: Vietnamese, English, German, French, and Mandarin Chinese. |
| Outcome: | The proposed model covers Vietnamese, English, German, French, and Mandarin Chinese, and is the first multilingual ASR dataset across five languages. |
TokenVerse: Towards Unifying Speech and NLP Tasks via Transducer-based ASR (2024.emnlp-main)
Copied to clipboard
Shashi Kumar, Srikanth Madikeri, Juan Pablo Zuluaga Gomez, Iuliia Thorbecke, Esaú Villatoro-tello, Sergio Burdisso, Petr Motlicek, Karthik S, Aravind Ganapathiraju
| Challenge: | Existing approaches to automatic speech recognition use cascaded pipelines for tasks like voice activity detection, diarization, transcription and subsequent processing. |
| Approach: | They propose a single Transducer-based model that integrates task-specific tokens into the reference text during ASR model training, streamlining inference and eliminating the need for separate NLP models. |
| Outcome: | The proposed model outperforms the existing pipeline on speaker change detection, endpointing, and NER tasks while outperforming the existing model in individual task performance. |
Dual-decoder Transformer for Joint Automatic Speech Recognition and Multilingual Speech Translation (2020.coling-main)
Copied to clipboard
| Challenge: | Existing models for automatic speech recognition and multilingual speech translation are on par with cascade counterparts. |
| Approach: | They propose a dual-decoder Transformer architecture that performs automatic speech recognition and multilingual speech translation. |
| Outcome: | The proposed models outperform the previously-reported highest translation performance in multilingual settings and bilingual one-to-one results. |
Comparison of Conventional Hybrid and CTC/Attention Decoders for Continuous Visual Speech Recognition (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent advances have been achieved in Visual Speech Recognition (VSR) despite the lack of data, there is no clear comparison between different types of decoders for certain languages and tasks. |
| Approach: | They focused on how the conventional DNN-HMM decoder behaves depending on the amount of data used for their estimation. |
| Outcome: | The proposed model improves the CTC/Attention model in data-scarcity scenarios while requiring less training time and fewer parameters. |
Variational Attention for Sequence-to-Sequence Models (C18-1)
Copied to clipboard
| Challenge: | Existing variational autoencoders encode data to latent variables and then decode them into target data. |
| Approach: | They propose a variational attention mechanism where the attention vector is also modeled as Gaussian distributed random variables. |
| Outcome: | The proposed method reduces the variational latent space bypassing phenomenon as it increases diversity of generated sentences. |
Attention as a Guide for Simultaneous Speech Translation (2023.acl-long)
Copied to clipboard
| Challenge: | EDAtt uses attention patterns to determine when to emit partial translations . results show that it yields better results compared to existing SimulST policies . |
| Approach: | They propose an adaptive policy that exploits attention patterns between audio source and target textual translation to guide an offline-trained ST model during simultaneous inference. |
| Outcome: | The proposed policy yields better results on en->de, compared to the current state of the art. |