Papers by Myle Ott
Facebook AI’s WAT19 Myanmar-English Translation Task Submission (D19-52)
Copied to clipboard
Peng-Jen Chen, Jiajun Shen, Matthew Le, Vishrav Chaudhary, Ahmed El-Kishky, Guillaume Wenzek, Myle Ott, Marc’Aurelio Ranzato
| Challenge: | Using back-translation, we can improve generalization by using noisy channel re-ranking and ensembling. |
| Approach: | They propose to use BPE-based transformer models to leverage monolingual data to improve generalization and use noisy channel re-ranking and ensembling to improve results. |
| Outcome: | The proposed system improves on the baseline system trained exclusively on the provided small parallel dataset, and the human evaluation and BLEU score are higher. |
Understanding Back-Translation at Scale (D18-1)
Copied to clipboard
| Challenge: | An effective method to improve neural machine translation with monolingual data is to augment the parallel training corpus with back-translations of target language sentences. |
| Approach: | They propose to augment parallel training corpus with back-translations of target language sentences to improve neural machine translation with monolingual data. |
| Outcome: | The proposed method achieves a state-of-the-art of 35 BLEU on the WMT’14 English-German test set. |
The FLORES Evaluation Datasets for Low-Resource Machine Translation: Nepali–English and Sinhala–English (D19-1)
Copied to clipboard
Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, Marc’Aurelio Ranzato
| Challenge: | a vast majority of language pairs in the world are considered low-resource because they have little parallel data available. |
| Approach: | They propose to use a dataset to evaluate methods trained on low-resource language pairs . they report baseline performance using supervised, weakly supervised and semi-supervised settings . |
| Outcome: | The proposed evaluation datasets show that current state-of-the-art methods perform poorly on this benchmark, posing a challenge to the research community working on low-resource MT. |
Efficient Large Scale Language Modeling with Mixtures of Experts (2022.emnlp-main)
Copied to clipboard
Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, Giridharan Anantharaman, Xian Li, Shuohui Chen, Halil Akin, Mandeep Baines, Louis Martin, Xing Zhou, Punit Singh Koura, Brian O’Horo, Jeffrey Wang, Luke Zettlemoyer, Mona Diab, Zornitsa Kozareva, Veselin Stoyanov
| Challenge: | Mixture of Experts layers (MoEs) enable efficient scaling of language models . large autoregressive language models such as GPT-3 can be adapted to a wide range of tasks . |
| Approach: | They propose to use Mixture of Experts layers to enable efficient scaling of language models . they find that MoEs are substantially more compute efficient than dense models compared to MoE models - but only when they are more modestly trained . |
| Outcome: | The proposed model outperforms dense models in a wide range of tasks and domains. |
Recipes for Building an Open-Domain Chatbot (2021.eacl-main)
Copied to clipboard
Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, Jason Weston
| Challenge: | Existing work shows that scaling models in the number of parameters and the size of the data they are trained on gives improved results, but other factors are important. |
| Approach: | They propose to build open-domain chatbots that can be scaled to improve their performance . they use a blend of cognitive and cognitive skills to build a model that combines these skills . |
| Outcome: | The proposed models outperform existing approaches in multi-turn dialogue on engagingness and humanness measurements. |
Few-shot Learning with Multilingual Generative Language Models (2022.emnlp-main)
Copied to clipboard
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, Xian Li
| Challenge: | Large-scale generative language models such as GPT-3 are competitive few-shot learners. |
| Approach: | They train multilingual generative language models on a corpus covering a diverse set of languages and study their few- and zero-shot learning capabilities. |
| Outcome: | The proposed model outperforms GPT-3 on 171 out of 182 directions with 32 training examples and surpasses the official supervised baseline in 45 directions. |
Unsupervised Cross-lingual Representation Learning at Scale (2020.acl-main)
Copied to clipboard
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, Veselin Stoyanov
| Challenge: | Pretraining multilingual language models at scale leads to performance gains for cross-lingual transfer tasks. |
| Approach: | They present a transformer-based multilingual masked language model pre-trained on 100 languages . they show that pretraining multilingual models at scale leads to significant performance gains . |
| Outcome: | The proposed model outperforms multilingual BERT (mBERT) on cross-lingual benchmarks. |
General Purpose Text Embeddings from Pre-trained Language Models for Scalable Inference (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Large pre-trained language models are currently used for many NLP tasks . however, inference for these models requires significant computational resources . |
| Approach: | They propose to use a shared text encoder to amortize the computational cost of inference over multiple tasks. |
| Outcome: | The proposed method reduces the size of the extracted representations by a factor of 16 to store them for later use. |
fairseq: A Fast, Extensible Toolkit for Sequence Modeling (N19-4)
Copied to clipboard
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, Michael Auli
| Challenge: | OpenNMT is a community-built toolkit written in multiple languages with an emphasis on extensibility. |
| Approach: | They propose to use PyTorch to train custom sequence models for translation, summarization, language modeling, and other tasks. |
| Outcome: | The proposed toolkit is fast, extensible, and useful for both research and production. |
How Decoding Strategies Affect the Verifiability of Generated Text (2020.findings-emnlp)
Copied to clipboard
Luca Massarelli, Fabio Petroni, Aleksandra Piktus, Myle Ott, Tim Rocktäschel, Vassilis Plachouras, Fabrizio Silvestri, Sebastian Riedel
| Challenge: | Recent advances in pre-trained language models have generated text of an increasingly high quality. |
| Approach: | They propose a decoding strategy that produces less repetitive and more verifiable text. |
| Outcome: | The proposed method produces less repetitive and more verifiable text than previously used decoding strategies. |
Phrase-Based & Neural Unsupervised Machine Translation (D18-1)
Copied to clipboard
| Challenge: | Recent advances in machine translation have reported near human-level performance on several languages, yet their effectiveness strongly relies on the availability of large amounts of parallel sentences. |
| Approach: | They propose two models that leverage a careful initialization of the parameters and denoising effect of language models. |
| Outcome: | The proposed models outperform the current methods on English-French and German-English benchmarks while being simpler and having fewer hyper-parameters. |
Analyzing the Forgetting Problem in Pretrain-Finetuning of Open-domain Dialogue Response Models (2021.eacl-main)
Copied to clipboard
| Challenge: | a large-scale unsupervised pretraining has been shown to greatly boost the performance of natural language processing models. |
| Approach: | They propose an intuitive finetuning strategy to regularize the finetune process . they propose a mix-review strategy to alleviate the forgetting problem . |
| Outcome: | The proposed strategy regularizes the finetuning process, and the forgetting problem is alleviated . the proposed strategy also improves the performance of the resulting model . |
On The Evaluation of Machine Translation Systems Trained With Back-Translation (2020.acl-main)
Copied to clipboard
| Challenge: | Back-translation is a data augmentation technique that can be used to improve neural machine translation systems. |
| Approach: | They propose to combine back-translation with a language model score to measure fluency. |
| Outcome: | The proposed method improves translation quality of natural text and translationese according to professional translators. |
Classical Structured Prediction Losses for Sequence to Sequence Learning (N18-1)
Copied to clipboard
| Challenge: | Recent work on training neural attention models at the sequence level has focused on a series of objective functions commonly used for structured prediction. |
| Approach: | They propose to use objective functions commonly used to train linear models for structured prediction to train neural attention models at the sequence-level using either reinforcement learning-style methods or beam search optimization. |
| Outcome: | The proposed model outperforms beam search optimization on German-English translation and abstractive summarization tasks. |
The Source-Target Domain Mismatch Problem in Machine Translation (2021.eacl-main)
Copied to clipboard
Jiajun Shen, Peng-Jen Chen, Matthew Le, Junxian He, Jiatao Gu, Myle Ott, Michael Auli, Marc’Aurelio Ranzato
| Challenge: | Despite the interconnected world we live in, people in different places talk about different things in different parts of the world. |
| Approach: | They propose a metric to quantify the effect of local context in machine translation and propose measurable results. |
| Outcome: | The proposed metric can be used to quantify the effect of local context on the use of language in machine translation systems on low resource languages. |