Papers by Ankur Bapna
Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference (2021.findings-emnlp)
Copied to clipboard
Sneha Kudugunta, Yanping Huang, Ankur Bapna, Maxim Krikun, Dmitry Lepikhin, Minh-Thang Luong, Orhan Firat
| Challenge: | Sparse Mixture-of-Experts (MoE) is a successful approach for scaling multilingual translation models to billions of parameters without a proportional increase in training computation. |
| Approach: | They propose to use a task-level routing approach to extract smaller, ready-to-deploy sub-networks from large sparse models by ignoring distillation. |
| Outcome: | Experiments on WMT and a web-scale dataset show that task-level routing outperforms token-level MoE models by +1.0 BLEU on average across 30 language pairs. |
Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus (2020.coling-main)
Copied to clipboard
| Challenge: | Large text corpora are increasingly important for a wide variety of NLP tasks. |
| Approach: | They propose to train automatic language identification models on up to 1,629 languages . they find that human-judged accuracy for web-crawl text corpora is only around 5% for many lower-resource languages. |
| Outcome: | The proposed models achieve over 90% average F1 on 1,629 languages . human-judged accuracy for web-crawl text corpora is only around 5% for many lower-resource languages - suggesting a need for more robust evaluation. |
Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets (2022.tacl-1)
Copied to clipboard
Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Papadimitriou, Salomey Osei, Pedro Ortiz Suarez, Iroro Orife, Kelechi Ogueji, Andre Niyongabo Rubungo, Toan Q. Nguyen, Mathias Müller, André Müller, Shamsuddeen Hassan Muhammad, Nanda Muhammad, Ayanda Mnyakeni, Jamshidbek Mirzakhalov, Tapiwanashe Matangira, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, Mathias Jenny, Orhan Firat, Bonaventure F. P. Dossou, Sakhile Dlamini, Nisansa de Silva, Sakine Çabuk Ballı, Stella Biderman, Alessia Battisti, Ahmed Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime, Ayodele Awokoya, Duygu Ataman, Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, Mofetoluwa Adeyemi
| Challenge: | Lower-resource corpora have systematic issues, including mislabeled or nonstandard/ambiguous language codes. |
| Approach: | They manually audit the quality of 205 language-specific corpora released with five major public datasets. |
| Outcome: | The results show that lower-resource corpora have systematic issues even for non-proficient speakers. |
The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation (P18-1)
Copied to clipboard
Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Mike Schuster, Noam Shazeer, Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Zhifeng Chen, Yonghui Wu, Macduff Hughes
| Challenge: | In recent years, the emergence of seq2seq models has revolutionized the field of machine translation by replacing traditional phrase-based approaches with neural machine translation (NMT) systems based on the encoder-decoder paradigm. |
| Approach: | They propose to use a convolutional seq2seq model to combine the strengths of the two approaches. |
| Outcome: | The proposed architectures outperform the existing models on the WMT’14 benchmark dataset. |
Investigating Multilingual NMT Representations at Scale (D19-1)
Copied to clipboard
| Challenge: | Multilingual Neural Machine Translation models have shown success in transfer learning settings, but their mode of transfer remains elusive. |
| Approach: | They propose to use a representation similarity framework to compare multilingual representations using a SVCCA representation similar to the previous work. |
| Outcome: | The proposed model can be used to compare representations across languages and layers. |
Simple, Scalable Adaptation for Neural Machine Translation (D19-1)
Copied to clipboard
| Challenge: | Recent advances in deep learning have led to significantly improved quality on Neural Machine Translation (NMT) however, performance on out-of-domain data or low resource languages remains poor. |
| Approach: | They propose a simple yet efficient approach for adapting pre-trained models to multiple tasks simultaneously. |
| Outcome: | The proposed approach is on par with full fine-tuning on domain adaptation and massively multilingual NMT on a massively multilingual dataset. |
Multilingual Mix: Example Interpolation Improves Multilingual Neural Machine Translation (2022.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to train multilingual models to learn the inductive bias of a shared vocabulary and set of parameters across languages. |
| Approach: | They propose to use a multilingual crossover encoder-decoder to fuse language pairs at an instance level to encourage sharing of input and output spaces. |
| Outcome: | The proposed approach improves quality on English-to-Many, Many-to English and zero-shot translation tasks from +0.5 BLEU up to +5.5 BLUE points. |
Multilingual Document-Level Translation Enables Zero-Shot Transfer From Sentences to Documents (2022.acl-long)
Copied to clipboard
| Challenge: | Document-level neural machine translation (DocNMT) is a powerful tool for integrating cross-sentence context into translations. |
| Approach: | They explore whether and how contextual modeling in DocNMT is transferable via multilingual modeling. |
| Outcome: | The proposed model can be used to transfer from teacher languages to student languages with no documents but sentence level data. |
Training Deeper Neural Machine Translation Models with Transparent Attention (D18-1)
Copied to clipboard
| Challenge: | Existing NMT models are shallow in comparison to convolutional models used for both text and vision tasks. |
| Approach: | They propose to modify the attention mechanism to ease the optimization of deeper models by a simple modification to the seq2seq with attention paradigm. |
| Outcome: | The proposed model achieves consistent gains of 0.7-1.1 BLEU on the benchmark WMT’14 English-German and WMT'15 Czech-English tasks. |
Revisiting Character-Based Neural Machine Translation with Capacity and Compression (D18-1)
Copied to clipboard
| Challenge: | Translating characters instead of words or word-fragments can simplify the processing pipeline but results in longer sequences . |
| Approach: | They propose to use sequence-to-sequence architectures of sufficient depth to solve the problem . they also evaluate the performance versus computation time tradeoffs they offer . |
| Outcome: | The proposed models outperform models operating over word fragments in character-level NMT, the authors show . they also show that the proposed models do not match the performance of their deep character baseline model . |
Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine Translation (2020.acl-main)
Copied to clipboard
Aditya Siddhant, Ankur Bapna, Yuan Cao, Orhan Firat, Mia Chen, Sneha Kudugunta, Naveen Arivazhagan, Yonghui Wu
| Challenge: | Existing multilingual NMT approaches do not utilize the abundance of monolingual data, especially in low-resource languages. |
| Approach: | They propose to combine monolingual data with self-supervision to pre-train translation models and fine-tune on small amounts of supervised data. |
| Outcome: | The proposed approach improves translation quality of low-resource languages and zero-shot translation quality. |
Non-Parametric Adaptation for Neural Machine Translation (N19-1)
Copied to clipboard
| Challenge: | Neural Networks trained with gradient descent are susceptible to catastrophic forgetting due to parameter shift during the training process. |
| Approach: | They propose a semi-parametric approach that relies on local phrase level similarities to retrieve neighboring phrases that are useful for translation even when overall sentence similarity is low. |
| Outcome: | The proposed approach performs well on a heterogeneous dataset with WMT, IWSLT, JRC-Acquis and OpenSubtitles. |