The Interpreter Understands Your Meaning: End-to-end Spoken Language Understanding Aided by Speech Translation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Modern artificial intelligence is characterized by large pretrained language models with strong language capabilities to be adapted to various downstream tasks. |
| Approach: | They propose to use the task of speech translation (ST) to pretrain speech models for end-to-end SLU on intra- and cross-lingual scenarios. |
| Outcome: | The proposed model achieves higher performance over baselines on monolingual and multilingual intent classification as well as spoken question answering using SLURP, MINDS-14, and NMSQA benchmarks. |
Similar Papers
Speech Translation and the End-to-End Promise: Taking Stock of Where We Are (2020.acl-main)
Copied to clipboard
| Challenge: | Until recently, the only feasible approach to translating acoustic speech signals into text was the cascaded approach. |
| Approach: | They propose a classification of the main challenges of traditional approaches to speech translation . they argue that end-to-end models fall short due to compromises made to address data scarcity . |
| Outcome: | This paper provides a brief survey of the main challenges of traditional approaches in speech translation . it reveals that many end-to-end models fail due to compromises made to address data scarcity. |
Tutorial: End-to-End Speech Translation (2021.eacl-tutorials)
Copied to clipboard
| Challenge: | Speech translation is the translation of speech in one language typically to text in another, traditionally accomplished through a combination of automatic speech recognition and machine translation. |
| Approach: | This tutorial introduces the techniques used in cutting-edge research on speech translation. |
| Outcome: | The proposed models achieve state-of-the-art performance with end-to-end speech translation for both high- and low-resource languages. |
Token-level Sequence Labeling for Spoken Language Understanding using Compositional End-to-End Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | End-to-end spoken language understanding systems model sequence labeling as a sequence prediction task causing a divergence from its well-established token-level tagging formulation. |
| Approach: | They propose to model sequence labeling as a sequence prediction task . their systems explicitly separate the added complexity of recognizing spoken mentions from the NLU task of sequence labelling . |
| Outcome: | The proposed systems outperform both cascaded and direct models on a labeling task of named entity recognition across SLU benchmarks. |
Curriculum Pre-training for End-to-End Speech Translation (2020.acl-main)
Copied to clipboard
| Challenge: | End-to-end speech translation requires a powerful encoder to transcribe, understand and learn cross-lingual semantics simultaneously. |
| Approach: | They propose a curriculum pre-training method that includes an elementary course for transcription learning and two advanced courses for understanding the utterance and mapping words in two languages. |
| Outcome: | The proposed method improves on En-De and En-Fr speech translation benchmarks. |
SLUE Phase-2: A Benchmark Suite of Diverse Spoken Language Understanding Tasks (2023.acl-long)
Copied to clipboard
Suwon Shon, Siddhant Arora, Chyi-Jiunn Lin, Ankita Pasad, Felix Wu, Roshan S Sharma, Wei-Lun Wu, Hung-yi Lee, Karen Livescu, Shinji Watanabe
| Challenge: | Spoken language understanding (SLU) tasks have received little attention and resources compared to lower-level tasks like speech and speaker recognition. |
| Approach: | They propose annotated SLU benchmark tasks based on freely available speech data to complement existing benchmarks and address gaps in the evaluation landscape. |
| Outcome: | The proposed benchmarks complement existing benchmarks and address gaps in the evaluation landscape. |
Simple and Effective Unsupervised Speech Translation (2023.acl-long)
Copied to clipboard
Changhan Wang, Hirofumi Inaguma, Peng-Jen Chen, Ilia Kulikov, Yun Tang, Wei-Ning Hsu, Michael Auli, Juan Pino
| Challenge: | Existing methods to train speech models without labeled data are limited for most languages. |
| Approach: | They propose a pipeline approach to build speech translation systems without labeled data by leveraging recent advances in unsupervised speech recognition, machine translation and speech synthesis. |
| Outcome: | The proposed approach outperforms the state-of-the-art in unsupervised speech recognition by 3.2 BLEU on the Libri-Trans benchmark and the best supervised end-to-end models from only two years ago by an average of 5.0 BLUE over five X-En directions. |
Integrating Pre-Trained Speech and Language Models for End-to-End Speech Recognition (2024.findings-acl)
Copied to clipboard
| Challenge: | Mainstream of automatic speech recognition (ASR) has shifted from pipeline methods to end-to-end (E2E) methods. |
| Approach: | They propose to integrate a pre-trained speech representation model and a large language model (LLM) for automatic speech recognition in an end-to-end manner. |
| Outcome: | The proposed model achieves comparable performance to modern E2E ASR models by utilizing powerful pre-training models with the proposed integrated approach. |
Cross-lingual Spoken Language Understanding with Regularized Representation Alignment (2020.emnlp-main)
Copied to clipboard
| Challenge: | despite promising results, current cross-lingual models suffer from imperfect cross-linguistic representation alignments between the source and target languages, which makes the performance sub-optimal. |
| Approach: | They propose a regularization approach to align word-level and sentence-level representations across languages without external resources. |
| Outcome: | The proposed model outperforms state-of-the-art models in few-shot and zero-shot scenarios and achieves comparable performance to supervised training with all training data. |
Can Machine Translation Bridge Multilingual Pretraining and Cross-lingual Transfer Learning? (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing models that pretrain for cross-lingual tasks do not improve cross-linguistic learning. |
| Approach: | They propose to employ machine translation as a continued training objective to enhance language representation learning by bridging multilingual pretraining and cross-lingual applications. |
| Outcome: | The proposed model performance is compared with existing models and their latent representations. |
Streaming Models for Joint Speech Recognition and Translation (2021.eacl-main)
Copied to clipboard
| Challenge: | Using end-to-end models for speech translation has become a focus of the ST community . cascaded models have the advantage of including automatic speech recognition output . |
| Approach: | They propose a model that condenses sound waves into translated text and integrates automatic speech recognition outputs into the models. |
| Outcome: | The proposed model is statistically similar to cascading models, but has half the number of parameters. |