| Challenge: | a multi-task learning approach is currently available for speech recognition and speech synthesis models . |
| Approach: | They propose a parameter-efficient approach to learning ASR and TTS jointly . they use English as a resource-rich language and Arabic as 'low-resource' language . |
| Outcome: | The proposed model saves 50% of computational and memory costs while learning ASR and TTS jointly. |
Similar Papers
Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Large-scale language models can be fine-tuned to learn highly transferable embedding, but they are expensive and require multiple model parameters. |
| Approach: | They propose a way to fine-tune multiple down-stream generation tasks simultaneously using a single, large pretrained model. |
| Outcome: | The proposed model can maintain or improve the performance of fine-tuning the whole model. |
Unified Speech-Text Pre-training for Speech Translation and Recognition (2022.acl-long)
Copied to clipboard
Yun Tang, Hongyu Gong, Ning Dong, Changhan Wang, Wei-Ning Hsu, Jiatao Gu, Alexei Baevski, Xian Li, Abdelrahman Mohamed, Michael Auli, Juan Pino
| Challenge: | Existing methods to pre-train speech and text use unlabeled data to learn universal feature representations. |
| Approach: | They propose a method to jointly pre-train speech and text in an encoder-decoder modeling framework for speech translation and recognition. |
| Outcome: | The proposed method achieves between 1.7 and 2.3 BLEU improvement above the state of the art on the MuST-C speech translation dataset and comparable WERs to wav2vec 2.0 on the Librispeech speech recognition task. |
HyperPELT: Unified Parameter-Efficient Language Model Tuning for Both Language and Vision-and-Language Tasks (2023.findings-acl)
Copied to clipboard
| Challenge: | Pretraining and fine-tuning are the dominant paradigms in natural language processing. |
| Approach: | They propose a parameter-efficient multitask learning framework that takes trainable hyper-embeddings and visual modality as input and outputs weights for different modules in a pretrained language model. |
| Outcome: | The proposed framework adds fewer trainable parameters in multi-task learning while achieving superior performances and transfer ability compared to state-of-the-art methods. |
Boosting Inference Efficiency: Unleashing the Power of Parameter-Shared Pre-trained Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Parameter-shared pre-trained language models (PLMs) have emerged as a successful approach in resource-constrained environments. |
| Approach: | They propose a method to enhance the inference efficiency of parameter-shared PLMs by pre-training models that can achieve even greater acceleration. |
| Outcome: | The proposed method improves inference efficiency on autoregressive and autoencoding models. |
A Survey of Multilingual Models for Automatic Speech Recognition (2022.lrec-1)
Copied to clipboard
| Challenge: | Automatic Speech Recognition (ASR) systems have achieved human-like performance for a few languages, but the majority of the world’s languages do not have usable systems due to the lack of large speech datasets to train these models. |
| Approach: | They propose to use unlabeled speech data to build multilingual ASR models that can be used for improved performance on low-resource languages. |
| Outcome: | The proposed models can be used to improve performance on low-resource languages by using unlabeled speech data. |
Hello, It’s GPT-2 - How Can I Help You? Towards the Use of Pretrained Language Models for Task-Oriented Dialogue Systems (D19-56)
Copied to clipboard
| Challenge: | Statistical conversational systems are complex, timeintensive, expensive, and not easily transferable due to data scarcity. |
| Approach: | They propose a task-oriented dialogue model that operates on text input . they validate it on multi-domain task-orientated dialogues from a multi-word dataset . |
| Outcome: | The proposed model bypasses explicit policy and language generation modules on multi-domain task-oriented dialogues from the MultiWOZ dataset. |
Massive End-to-end Speech Recognition Models with Time Reduction (2024.naacl-long)
Copied to clipboard
Weiran Wang, Rohit Prabhavalkar, Haozhe Shan, Zhong Meng, Dongseong Hwang, Qiujia Li, Khe Chai Sim, Bo Li, James Qin, Xingyu Cai, Adam Stooke, Chengjian Zheng, Yanzhang He, Tara Sainath, Pedro Moreno Mengibar
| Challenge: | Using the neural architecture of Google’s universal speech model, we reduce the frame rate and speed up training and inference. |
| Approach: | They propose to use the neural architecture of Google’s universal speech model with additional funnel pooling layers to significantly reduce the frame rate and speed up training and inference. |
| Outcome: | The proposed methods work with both connectionist temporal classification (CTC) and RNN-Transducer (RNN-T) and over two domains. |
Rethinking and Improving Multi-task Learning for End-to-end Speech Translation (2023.emnlp-main)
Copied to clipboard
| Challenge: | auxiliary tasks are highly consistent with end-to-end speech translation (ST) but their effectiveness has not been thoroughly studied. |
| Approach: | They propose an improved multi-task learning approach for the ST task that bridges the modal gap by mitigating the difference in length and representation. |
| Outcome: | The proposed approach achieves state-of-the-art on the MuST-C dataset with 20.8% of training time required by the current SOTA method. |
Tied Multitask Learning for Neural Speech Translation (N18-1)
Copied to clipboard
| Challenge: | Recent efforts in endangered language documentation focus on collecting spoken language resources . BULB project uses mobile app to collect spoken resources accompanied by spoken translations . |
| Approach: | They propose a model where the second task decoder receives information from the first task . they apply regularization that encourages transitivity and invertibility to the model . |
| Outcome: | The proposed model improves performance on low-resource speech transcription and translation tasks. |
Leveraging Large Pre-trained Multilingual Models for High-Quality Speech-to-Text Translation on Industry Scenarios (2025.coling-main)
Copied to clipboard
| Challenge: | Speech-to-Text Translation systems rely on a sequential pipeline that combines ASR and MT models. |
| Approach: | They propose a parameter-efficient framework that integrates one LPSM with a multilingual MT engine. |
| Outcome: | The proposed framework integrates one LPSM with a multilingual MT engine. |