Sparse Mixers: Combining MoE and Mixing to build a more efficient BERT (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Sparse Mixer encoder model outperforms BERT on GLUE and SuperGLUE, trains 65% faster and runs inference 61% faster. |
| Approach: | They combine the capacity of sparsely gated Mixture-of-Experts (MoE) with the speed and stability of linear, mixing transformations to design the Sparse Mixer encoder model. |
| Outcome: | The proposed model outperforms BERT on GLUE and SuperGLUE but trains and runs twice as fast. |
Similar Papers
Fixing MoE Over-Fitting on Low-Resource Languages in Multilingual Machine Translation (2023.findings-acl)
Copied to clipboard
| Challenge: | Sparsely gated Mixture of Experts (MoE) models are a compute-efficient method to scale model capacity for multilingual machine translation tasks. |
| Approach: | They propose a regularization strategy that prevents over-fitting of MoE models on low-resource tasks and conditional MoE Routing and curriculum learning methods that prevent over- fitting. |
| Outcome: | The proposed methods improve the performance of MoE models on low-resource tasks without adversely affecting high-res tasks. |
A Closer Look into Mixture-of-Experts in Large Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Mixture-of-experts (MoE) architectures are gaining increasing attention for their unique properties and remarkable performance. |
| Approach: | They propose a mixture-of-experts architecture that allows for model scaling without sacrificing computational efficiency. |
| Outcome: | The proposed model increases model size without sacrificing computational efficiency . the proposed model is modular and can be used by a broad spectrum of practitioners . |
Bag of Tricks for Sparse Mixture-of-Experts: A Benchmark Across Reasoning, Efficiency, and Safety (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing benchmarks focus on isolated aspects of MoE, with conflicting conclusions . a lack of consensus on optimal design choices is limiting to specific aspects of the model. |
| Approach: | They propose to evaluate two popular MoE backbones across four dimensions of design choices . they find token-level routing and z-loss regularization improve reasoning performance . |
| Outcome: | The proposed framework evaluates two popular MoE backbones on over eight metrics. |
XMoE: Sparse Models with Fine-grained and Adaptive Expert Selection (2024.findings-acl)
Copied to clipboard
| Challenge: | XMoE leverages small experts and a threshold-based router to selectively engage only essential parameters. |
| Approach: | They propose a novel MoE that leverages small experts to selectively engage only essential parameters. |
| Outcome: | The proposed model can reduce computation load at MoE layers by over 50% without sacrificing performance. |
CLIP-UP: A Simple and Efficient Mixture-of-Experts CLIP Training Recipe with Sparse Upcycling (2025.findings-emnlp)
Copied to clipboard
Xinze Wang, Chen Chen, Yinfei Yang, Hong-You Chen, Bowen Zhang, Aditya Pal, Xiangxin Zhu, Xianzhi Du
| Challenge: | Mixture-of-Experts (MoE) models are crucial for scaling model capacity while controlling inference costs. |
| Approach: | They propose an alternative training strategy that converts a dense CLIP model into a sparse MoE architecture. |
| Outcome: | The proposed training strategy outperforms dense models on COCO and Flickr30k benchmarks. |
Scaling Vision-Language Models with Sparse Mixture of Experts (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a study explores the effectiveness of mixture-of-experts (MoE) techniques in scaling vision-language models . alayrac and colleagues demonstrate the effectiveness and performance of MoE in scaling VLMs . |
| Approach: | They propose to use sparsely-gated mixture-of-experts techniques to scale vision-language models . they show that MoE can achieve state-of the-art performance over dense models a range of benchmarks . |
| Outcome: | The proposed approach achieves state-of-the-art performance over dense models of equivalent computational cost. |
Faster MoE LLM Inference for Extremely Large Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing inference optimizations for coarse-grained Mixture-of-Experts models implicitly assume a fixed activation budget, which is poorly understood. |
| Approach: | They propose a training-free policy that adapts token-level activation using router confidence and entropy while remaining within the model’s original budget. |
| Outcome: | The proposed skipping policy can provide substantial throughput gains, but optimal static schedules vary significantly across models and routing mechanisms. |
Efficiently Editing Mixture-of-Experts Models with Compressed Experts (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Mixture-of-Experts models allow for efficient scaling of large language models . fewer experts reduce computational costs, while more experts improve performance . |
| Approach: | They propose to activate only a subset of experts during training and inference . they propose compressed experts that preserve the most important experts . |
| Outcome: | The proposed approach preserves the most important experts while replacing other auxiliary activated experts with compressed experts. |
Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated considerable proficiency in general natural language processing tasks. |
| Approach: | They propose a parameter-efficient sparsity crafting method which crafts dense models into sparse models using the mixture-of-experts architecture. |
| Outcome: | The proposed method significantly reduces computational costs and GPU memory requirements, while maintaining the quality of approximation in function space. |
Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference (2021.findings-emnlp)
Copied to clipboard
Sneha Kudugunta, Yanping Huang, Ankur Bapna, Maxim Krikun, Dmitry Lepikhin, Minh-Thang Luong, Orhan Firat
| Challenge: | Sparse Mixture-of-Experts (MoE) is a successful approach for scaling multilingual translation models to billions of parameters without a proportional increase in training computation. |
| Approach: | They propose to use a task-level routing approach to extract smaller, ready-to-deploy sub-networks from large sparse models by ignoring distillation. |
| Outcome: | Experiments on WMT and a web-scale dataset show that task-level routing outperforms token-level MoE models by +1.0 BLEU on average across 30 language pairs. |