Papers by Changhan Wang
Fairseq S2T: Fast Speech-to-Text Modeling with Fairseq (2020.aacl-demo)
Copied to clipboard
| Challenge: | End-to-end sequence-to sequence (S2S) modeling has witnessed rapid growth in speech-totext (ST) tasks. |
| Approach: | They introduce fairseq S2T, a fairsq extension for speech-to-text modeling tasks such as end-to end speech recognition and speech-text translation. |
| Outcome: | The proposed extension provides end-to-end workflows from data pre-processing, model training to offline (online) inference. |
Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task (2021.acl-long)
Copied to clipboard
| Challenge: | Pretraining and multitask learning are widely used to improve the speech translation performance. |
| Approach: | They propose to train a speech translation model along with an auxiliary text translation task. |
| Outcome: | The proposed method improves translation quality by more than 2 BLEU over a strong baseline and achieves state-of-the-art results on the MuST-C English-German, English-French and English-Spanish language pairs. |
Multilingual Speech Translation from Efficient Finetuning of Pretrained Models (2021.acl-long)
Copied to clipboard
Xian Li, Changhan Wang, Yun Tang, Chau Tran, Yuqing Tang, Juan Pino, Alexei Baevski, Alexis Conneau, Michael Auli
| Challenge: | Recent advances in text pretraining and finetuning have improved multitasking applications significantly. |
| Approach: | They propose a minimalistic LNA finetuning approach to build multilingual speech-to-text translation using a pretrained speech encoder and text decoder. |
| Outcome: | The proposed approach surpasses the cascaded ST benchmark for 36 translation directions on the large-scale multilingual ST benchmark CoVoST 2. |
Dynamic Meta-Embeddings for Improved Sentence Representations (D18-1)
Copied to clipboard
| Challenge: | A sprawling literature has emerged about what word embeddings are most useful for which tasks . word embed-ding is a technique that can be used to learn word-level meaning representations for a variety of tasks. |
| Approach: | They propose a method for supervised learning of embedding ensembles that leads to state-of-the-art performance on a variety of tasks. |
| Outcome: | The proposed method leads to state-of-the-art performance on a variety of tasks. |
SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations (2023.acl-long)
Copied to clipboard
Paul-Ambroise Duquenne, Hongyu Gong, Ning Dong, Jingfei Du, Ann Lee, Vedanuj Goswami, Changhan Wang, Juan Pino, Benoît Sagot, Holger Schwenk
| Challenge: | SpeechMatrix is a large-scale multilingual corpus of speech-to-speech translations mined from real speech of European Parliament recordings. |
| Approach: | They present a large-scale multilingual corpus of speech-to-speech translations mined from real speech of European Parliament recordings. |
| Outcome: | The proposed model can train bilingual models on 136 language pairs with 418 thousand hours of speech. |
fairseq Sˆ2: A Scalable and Integrable Speech Synthesis Toolkit (2021.emnlp-demo)
Copied to clipboard
| Challenge: | Speech synthesis is the task of generating speech waveforms with desired characteristics, including but not limited to textual content, speaker identity, and speaking styles. |
| Approach: | They propose a fairseq extension for speech synthesis that implements autoregressive and non-AR text-to-speech models and their multi-speaker variants. |
| Outcome: | The proposed extension can train autoregressive and non-AR models and their multi-speaker variants with less curated data and has automatic metrics to facilitate faster iteration and analysis. |
VizSeq: a visual analysis toolkit for text generation tasks (D19-3)
Copied to clipboard
| Challenge: | Several softwares for text evaluation are available that do not provide detailed examples. |
| Approach: | They propose a visual analysis toolkit for instance-level and corpus-level system evaluation on a wide variety of text generation tasks. |
| Outcome: | The proposed toolkit covers most common n-gram metrics and latest embedding-based metrics such as BERTScore. |
Speech-to-Speech Translation for a Real-world Unwritten Language (2023.findings-acl)
Copied to clipboard
Peng-Jen Chen, Kevin Tran, Yilin Yang, Jingfei Du, Justine Kao, Yu-An Chung, Paden Tomasello, Paul-Ambroise Duquenne, Holger Schwenk, Hongyu Gong, Hirofumi Inaguma, Sravya Popuri, Changhan Wang, Juan Pino, Wei-Ning Hsu, Ann Lee
| Challenge: | a new study examines speech-to-speech translation (S2ST) that translates speech from one language into another . the research area for unwritten languages remains a research area with little exploration due to the lack of training data. |
| Approach: | They propose a system that translates speech from one language into another . they use Taiwanese Hokkien as an example of an unwritten language . |
| Outcome: | The proposed system can be used to train models in languages without standard writing systems. |
UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Units (2023.acl-long)
Copied to clipboard
Hirofumi Inaguma, Sravya Popuri, Ilia Kulikov, Peng-Jen Chen, Changhan Wang, Yu-An Chung, Yun Tang, Ann Lee, Shinji Watanabe, Juan Pino
| Challenge: | Experimental evaluations show that UnitY outperforms a single-pass speech-to-unit translation model by 2.5-4.2 ASR-BLEU with 2.83x decoding speed-up. |
| Approach: | They propose a two-pass direct S2ST architecture which generates textual representations and predicts discrete acoustic units . they show that UnitY outperforms a single-pass speech-to-unit translation model by 2.5-4.2 ASR-BLEU with 2.83x decoding speed-up. |
| Outcome: | The proposed architecture outperforms a single-pass speech-to-unit translation model by 2.5-4.2 ASR-BLEU with 2.83x decoding speed-up on large datasets. |
VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation (2021.acl-long)
Copied to clipboard
Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, Emmanuel Dupoux
| Challenge: | VoxPopuli provides 400K hours of unlabeled speech data in 23 languages . large amounts of multilingual audio data are needed to achieve similar progress for multilingual ASR and ST. |
| Approach: | They propose a large-scale multilingual corpus that provides 400K hours of unlabeled speech data in 23 languages. |
| Outcome: | The proposed corpus provides 400K hours of unlabeled speech data in 23 languages and 1.8K hours transcribed speeches in 15 languages and their aligned oral interpretations into 15 target languages totaling 17.3K hours. |
Textless Speech-to-Speech Translation on Real Data (2022.naacl-main)
Copied to clipboard
Ann Lee, Hongyu Gong, Paul-Ambroise Duquenne, Holger Schwenk, Peng-Jen Chen, Changhan Wang, Sravya Popuri, Yossi Adi, Juan Pino, Jiatao Gu, Wei-Ning Hsu
| Challenge: | Existing text-based speech-to-speech translation systems rely on cascaded approach . text-to text translation systems require text generation and a single input to generate output . |
| Approach: | They propose a textless speech-to-speech translation system that can translate speech from one language into another without the need of text data. |
| Outcome: | The proposed system can translate speech from one language into another without text data. |
Direct Speech-to-Speech Translation With Discrete Units (2022.acl-long)
Copied to clipboard
Ann Lee, Peng-Jen Chen, Changhan Wang, Jiatao Gu, Sravya Popuri, Xutai Ma, Adam Polyak, Yossi Adi, Qing He, Yun Tang, Juan Pino, Wei-Ning Hsu
| Challenge: | Existing direct speech-to-speech translation models rely on text generation as an intermediate step. |
| Approach: | They propose a direct speech-to-speech translation model that translates speech from one language to another without relying on intermediate text generation. |
| Outcome: | The proposed model produces 6.7 BLEUs in the Fisher Spanish-English dataset when trained without any text transcripts and with text supervision. |
Lightweight Adapter Tuning for Multilingual Speech Translation (2021.acl-short)
Copied to clipboard
| Challenge: | Adapter tuning is an efficient alternative to fine-tuning in NLP . a multilingual model could be outperformed by its bilingual counterparts . |
| Approach: | They propose to use adapter tuning to optimize for multilingual speech translation . they use pre-trained models to freeze pre-train parameters and inject lightweight modules . |
| Outcome: | The proposed adapters can specialize to specific language pairs with low extra cost . the proposed models outperform bilingual models on high-resource language pairs . |
Introducing Semantics into Speech Encoders (2023.acl-long)
Copied to clipboard
Derek Xu, Shuyan Dong, Changhan Wang, Suyoun Kim, Zhaojiang Lin, Bing Liu, Akshat Shrivastava, Shang-Wen Li, Liang-Hsuan Tseng, Guan-Ting Lin, Alexei Baevski, Hung-yi Lee, Yizhou Sun, Wei Wang
| Challenge: | Existing self-supervised speech encoders contain primarily acoustic rather than semantic information. |
| Approach: | They propose a task-agnostic unsupervised way to incorporate semantic information from large language model (LLM) systems into self-supervised speech encoders without labeled audio transcriptions. |
| Outcome: | The proposed approach improves spoken language understanding (SLU) performance by over 5% on intent classification (IC), with modest gains in named entity resolution (NER) and slot filling (SF), and spoken question answering (SQA) score by over 22%. |
CoVoST: A Diverse Multilingual Speech-To-Text Translation Corpus (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing datasets involve language pairs with English as source language, are low resource or lack labeled data. |
| Approach: | They propose a multilingual speech-to-text translation corpus from 11 languages into English . they provide empirical evidence of the quality of the data and provide initial benchmarks . |
| Outcome: | The proposed model is the first end-to-end multilingual model for spoken language translation. |
XLAVS-R: Cross-Lingual Audio-Visual Speech Representation Learning for Noise-Robust Speech Perception (2024.acl-long)
Copied to clipboard
| Challenge: | Speech recognition and translation systems perform poorly on noisy inputs, which are frequent in realistic environments. |
| Approach: | They propose a cross-lingual audio-visual speech representation model for noise-robust speech recognition and translation in over 100 languages. |
| Outcome: | The proposed model outperforms the previous state-of-the-art by 18.5% WER and 4.7 BLEU on downstream audio-visual speech recognition and translation tasks. |
SIMULEVAL: An Evaluation Toolkit for Simultaneous Translation (2020.emnlp-demos)
Copied to clipboard
| Challenge: | SimulEval is an evaluation toolkit for simultaneous text and speech translation. |
| Approach: | They propose a server-client scheme for simultaneous translation that uses server input and client policies to evaluate models. |
| Outcome: | The proposed evaluation toolkit is available for both text and speech translation. |
Simple and Effective Unsupervised Speech Translation (2023.acl-long)
Copied to clipboard
Changhan Wang, Hirofumi Inaguma, Peng-Jen Chen, Ilia Kulikov, Yun Tang, Wei-Ning Hsu, Michael Auli, Juan Pino
| Challenge: | Existing methods to train speech models without labeled data are limited for most languages. |
| Approach: | They propose a pipeline approach to build speech translation systems without labeled data by leveraging recent advances in unsupervised speech recognition, machine translation and speech synthesis. |
| Outcome: | The proposed approach outperforms the state-of-the-art in unsupervised speech recognition by 3.2 BLEU on the Libri-Trans benchmark and the best supervised end-to-end models from only two years ago by an average of 5.0 BLUE over five X-En directions. |
Unified Speech-Text Pre-training for Speech Translation and Recognition (2022.acl-long)
Copied to clipboard
Yun Tang, Hongyu Gong, Ning Dong, Changhan Wang, Wei-Ning Hsu, Jiatao Gu, Alexei Baevski, Xian Li, Abdelrahman Mohamed, Michael Auli, Juan Pino
| Challenge: | Existing methods to pre-train speech and text use unlabeled data to learn universal feature representations. |
| Approach: | They propose a method to jointly pre-train speech and text in an encoder-decoder modeling framework for speech translation and recognition. |
| Outcome: | The proposed method achieves between 1.7 and 2.3 BLEU improvement above the state of the art on the MuST-C speech translation dataset and comparable WERs to wav2vec 2.0 on the Librispeech speech recognition task. |
Dual-decoder Transformer for Joint Automatic Speech Recognition and Multilingual Speech Translation (2020.coling-main)
Copied to clipboard
| Challenge: | Existing models for automatic speech recognition and multilingual speech translation are on par with cascade counterparts. |
| Approach: | They propose a dual-decoder Transformer architecture that performs automatic speech recognition and multilingual speech translation. |
| Outcome: | The proposed models outperform the previously-reported highest translation performance in multilingual settings and bilingual one-to-one results. |