Papers by Dmitry Lepikhin
Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference (2021.findings-emnlp)
Copied to clipboard
Sneha Kudugunta, Yanping Huang, Ankur Bapna, Maxim Krikun, Dmitry Lepikhin, Minh-Thang Luong, Orhan Firat
| Challenge: | Sparse Mixture-of-Experts (MoE) is a successful approach for scaling multilingual translation models to billions of parameters without a proportional increase in training computation. |
| Approach: | They propose to use a task-level routing approach to extract smaller, ready-to-deploy sub-networks from large sparse models by ignoring distillation. |
| Outcome: | Experiments on WMT and a web-scale dataset show that task-level routing outperforms token-level MoE models by +1.0 BLEU on average across 30 language pairs. |