Papers by Michael Auli
Simple and Effective Noisy Channel Modeling for Neural Machine Translation (D19-1)
Copied to clipboard
| Challenge: | Previous work on noisy channel modeling relied on latent variable models that incrementally process the source and target sentence. |
| Approach: | They propose to use a standard sequence to sequence model which utilizes the entire source and target sentences to estimate posterior probability of a target sequence y given a source sequence x. |
| Outcome: | The proposed model outperforms direct models on German-English translations by up to 3.2 BLEU on four language pairs. |
Multilingual Speech Translation from Efficient Finetuning of Pretrained Models (2021.acl-long)
Copied to clipboard
Xian Li, Changhan Wang, Yun Tang, Chau Tran, Yuqing Tang, Juan Pino, Alexei Baevski, Alexis Conneau, Michael Auli
| Challenge: | Recent advances in text pretraining and finetuning have improved multitasking applications significantly. |
| Approach: | They propose a minimalistic LNA finetuning approach to build multilingual speech-to-text translation using a pretrained speech encoder and text decoder. |
| Outcome: | The proposed approach surpasses the cascaded ST benchmark for 36 translation directions on the large-scale multilingual ST benchmark CoVoST 2. |
Understanding Back-Translation at Scale (D18-1)
Copied to clipboard
| Challenge: | An effective method to improve neural machine translation with monolingual data is to augment the parallel training corpus with back-translations of target language sentences. |
| Approach: | They propose to augment parallel training corpus with back-translations of target language sentences to improve neural machine translation with monolingual data. |
| Outcome: | The proposed method achieves a state-of-the-art of 35 BLEU on the WMT’14 English-German test set. |
Self-training Improves Pre-training for Natural Language Understanding (2021.naacl-main)
Copied to clipboard
Jingfei Du, Edouard Grave, Beliz Gunel, Vishrav Chaudhary, Onur Celebi, Michael Auli, Veselin Stoyanov, Alexis Conneau
| Challenge: | Unsupervised pretraining has led to improvements in natural language understanding . a data augmentation method can be used to generate labels for unlabeled examples . |
| Approach: | They propose a semi-supervised method which uses unlabeled data to retrieve sentences from a database of billions of unlabed sentences crawled from the web. |
| Outcome: | The proposed method improves on standard text classification benchmarks by 2.6% and knowledge distillation by few shots. |
QuickEdit: Editing Text & Translations by Crossing Words Out (N18-1)
Copied to clipboard
| Challenge: | Using statistical learning, a computer can rephrase a sentence by only pointing at words that should be avoided. |
| Approach: | They propose a framework for computer-assisted text editing that relies on simple interactions between human editors and tokens. |
| Outcome: | The proposed framework allows to get substantial modifications to a sentence without human intervention. |
fairseq: A Fast, Extensible Toolkit for Sequence Modeling (N19-4)
Copied to clipboard
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, Michael Auli
| Challenge: | OpenNMT is a community-built toolkit written in multiple languages with an emphasis on extensibility. |
| Approach: | They propose to use PyTorch to train custom sequence models for translation, summarization, language modeling, and other tasks. |
| Outcome: | The proposed toolkit is fast, extensible, and useful for both research and production. |
Pre-trained language model representations for language generation (N19-1)
Copied to clipboard
| Challenge: | Pre-trained language model representations have been successful in a wide range of language understanding tasks. |
| Approach: | They propose to use pre-trained language model representations to integrate them into sequence to sequence models and apply it to machine translation and abstractive summarization. |
| Outcome: | The proposed model is able to perform 5.3 BLEU in machine translation and 5.3 on the full text version of CNN/DailyMail. |
On The Evaluation of Machine Translation Systems Trained With Back-Translation (2020.acl-main)
Copied to clipboard
| Challenge: | Back-translation is a data augmentation technique that can be used to improve neural machine translation systems. |
| Approach: | They propose to combine back-translation with a language model score to measure fluency. |
| Outcome: | The proposed method improves translation quality of natural text and translationese according to professional translators. |
Classical Structured Prediction Losses for Sequence to Sequence Learning (N18-1)
Copied to clipboard
| Challenge: | Recent work on training neural attention models at the sequence level has focused on a series of objective functions commonly used for structured prediction. |
| Approach: | They propose to use objective functions commonly used to train linear models for structured prediction to train neural attention models at the sequence-level using either reinforcement learning-style methods or beam search optimization. |
| Outcome: | The proposed model outperforms beam search optimization on German-English translation and abstractive summarization tasks. |
Discriminative Reranking for Neural Machine Translation (2021.acl-long)
Copied to clipboard
| Challenge: | reranking models allow the integration of rich features to select a better output hypothesis within an n-best list or lattice. |
| Approach: | They use discriminative reranking to train a large transformer architecture to train an ranked list of hypotheses. |
| Outcome: | Experiments on four WMT directions show that discriminative reranking improves translation quality. |
The Source-Target Domain Mismatch Problem in Machine Translation (2021.eacl-main)
Copied to clipboard
Jiajun Shen, Peng-Jen Chen, Matthew Le, Junxian He, Jiatao Gu, Myle Ott, Michael Auli, Marc’Aurelio Ranzato
| Challenge: | Despite the interconnected world we live in, people in different places talk about different things in different parts of the world. |
| Approach: | They propose a metric to quantify the effect of local context in machine translation and propose measurable results. |
| Outcome: | The proposed metric can be used to quantify the effect of local context on the use of language in machine translation systems on low resource languages. |
Reservoir Transformers (2021.acl-long)
Copied to clipboard
| Challenge: | Using random initialization, we show that some transformers obtain impressive performance even when some of the layers are frozen. |
| Approach: | They propose to freeze transformer layers and use them to improve performance . they find that the transformers obtain impressive performance even when some of the layers are randomly initialized and never updated. |
| Outcome: | The proposed model improves on translation and language modelling tasks even when some layers are frozen. |
Simple and Effective Unsupervised Speech Translation (2023.acl-long)
Copied to clipboard
Changhan Wang, Hirofumi Inaguma, Peng-Jen Chen, Ilia Kulikov, Yun Tang, Wei-Ning Hsu, Michael Auli, Juan Pino
| Challenge: | Existing methods to train speech models without labeled data are limited for most languages. |
| Approach: | They propose a pipeline approach to build speech translation systems without labeled data by leveraging recent advances in unsupervised speech recognition, machine translation and speech synthesis. |
| Outcome: | The proposed approach outperforms the state-of-the-art in unsupervised speech recognition by 3.2 BLEU on the Libri-Trans benchmark and the best supervised end-to-end models from only two years ago by an average of 5.0 BLUE over five X-En directions. |
Cloze-driven Pretraining of Self-attention Networks (D19-1)
Copied to clipboard
| Challenge: | Existing work on pretraining language models has used unidirectional (left-to-right) or bi-directional (both left-to right and right-to left) LMs with loss function. |
| Approach: | They propose a bi-directional transformer model that pretrains both directions of a large language-model-inspired self-attention cloze model and propose clozing to predict each word in the training data. |
| Outcome: | The proposed model performs well on GLUE and state of the art benchmarks consistent with BERT. |
ELI5: Long Form Question Answering (P19-1)
Copied to clipboard
| Challenge: | Existing question answering datasets provide extractive or short answers, but less attention has been paid to open-ended questions that require explanations. |
| Approach: | They present a large-scale corpus for long form question answering . they use a Reddit forum to provide elaborate answers to open-ended questions . |
| Outcome: | The proposed model outperforms Seq2Seq, language modeling, and other models in human evaluations. |
Unified Speech-Text Pre-training for Speech Translation and Recognition (2022.acl-long)
Copied to clipboard
Yun Tang, Hongyu Gong, Ning Dong, Changhan Wang, Wei-Ning Hsu, Jiatao Gu, Alexei Baevski, Xian Li, Abdelrahman Mohamed, Michael Auli, Juan Pino
| Challenge: | Existing methods to pre-train speech and text use unlabeled data to learn universal feature representations. |
| Approach: | They propose a method to jointly pre-train speech and text in an encoder-decoder modeling framework for speech translation and recognition. |
| Outcome: | The proposed method achieves between 1.7 and 2.3 BLEU improvement above the state of the art on the MuST-C speech translation dataset and comparable WERs to wav2vec 2.0 on the Librispeech speech recognition task. |
Toward Joint Language Modeling for Speech Units and Text (2023.findings-emnlp)
Copied to clipboard
Ju-Chieh Chou, Chung-Ming Chien, Wei-Ning Hsu, Karen Livescu, Arun Babu, Alexis Conneau, Alexei Baevski, Michael Auli
| Challenge: | Speech and text are two major forms of human language and little effort has been made to model them together. |
| Approach: | They propose to combine speech and text models to create mixed speech-text data by using different tokenizers and automatic metrics to evaluate how well the model mixes speech and texts. |
| Outcome: | The proposed model improves over a speech-only baseline and shows zero-shot cross-modal transferability. |