SliceMoE: Routing Embedding Slices Instead of Tokens for Fine-Grained and Balanced Transformer Scaling (2025.emnlp-main)
Copied to clipboard
| Challenge: | Token-level routing assigns an entire semantic spectrum to each expert, creating capacity bottlenecks, load-balancing pathologies, and limited specialisation. |
| Approach: | They propose an architecture that routes contiguous slices of a token’s hidden vector and a lightweight shared router predicts the top-k experts. |
| Outcome: | The proposed architecture achieves 1.7x faster inference than dense baselines, 12–18% lower perplexity than parameter-matched token-MoE, and improved expert balance. |
Similar Papers
DSMoE: Matrix-Partitioned Experts with Dynamic Routing for Computation-Efficient Dense LLMs (2025.emnlp-main)
Copied to clipboard
Minxuan Lv, Zhenpeng Su, Leiyu Pan, Yizhe Xiong, Zijia Lin, Hui Chen, Wei Zhou, Jungong Han, Guiguang Ding, Wenwu Ou, Di Zhang, Kun Gai, Songlin Hu
| Challenge: | Existing sparsification methods like pruning can lose model knowledge through parameter removal. |
| Approach: | They propose a novel approach that achieves sparsification by partitioning pre-trained FFN layers into computational blocks. |
| Outcome: | The proposed approach achieves superior performance across language modeling and downstream tasks under equivalent computational constraints. |
XMoE: Sparse Models with Fine-grained and Adaptive Expert Selection (2024.findings-acl)
Copied to clipboard
| Challenge: | XMoE leverages small experts and a threshold-based router to selectively engage only essential parameters. |
| Approach: | They propose a novel MoE that leverages small experts to selectively engage only essential parameters. |
| Outcome: | The proposed model can reduce computation load at MoE layers by over 50% without sacrificing performance. |
StableMoE: Stable Routing Strategy for Mixture of Experts (2022.acl-long)
Copied to clipboard
| Challenge: | Existing learning-to-route methods suffer from the routing fluctuation issue . with the model scale growing, training speed will go slower and memory requirements are heavy . |
| Approach: | They propose a Mixture-of-Experts technique that can scale up the model size of Transformers with an affordable computational overhead. |
| Outcome: | The proposed method outperforms existing learning-to-route methods on language modeling and multilingual machine translation. |
FoldMoE: Efficient Long Sequence MoE Training via Attention-MoE Pipelining (2025.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to training LLMs with Mixture-of-Experts (MoE) architecture on long sequences are limited by the insufficient computation. |
| Approach: | They propose a MoE training system that enables token-level overlapping across entire Transformer blocks through novel attention-MoE pipelining. |
| Outcome: | The proposed system achieves 1.49x and 2.72x speedup over state-of-the-art token-level overlapping and non-overlapping baselines on GPT-MoE models with sequences up to 32K tokens. |
AutoMoE: Heterogeneous Mixture-of-Experts with Adaptive Computation for Efficient Neural Machine Translation (2023.findings-acl)
Copied to clipboard
Ganesh Jawahar, Subhabrata Mukherjee, Xiaodong Liu, Young Jin Kim, Muhammad Abdul-Mageed, Laks Lakshmanan, V.S., Ahmed Hassan Awadallah, Sebastien Bubeck, Jianfeng Gao
| Challenge: | Existing MoE designs do not consider computational constraints (e.g., FLOPs, latency) Existing works in MoE consider homogeneous design where the same number of experts of the same size are placed uniformly throughout the network. |
| Approach: | They propose a framework for designing heterogeneous MoEs under computational constraints. |
| Outcome: | The proposed framework achieves 4x inference speedup and FLOPs reduction over manual models and within 1 BLEU point of MoE SwitchTransformer over benchmark datasets for NMT. |
Mixture of Heterogeneous Grouped Experts for Language Modeling (2026.acl-industry)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) based on Mixture-of-Experts (MoE) enforce uniform expert sizes, creating a rigidity that fails to align computational costs with varying token-level complexity. |
| Approach: | They propose a mixture of heterogeneous grouped experts (MoHGE) that allows for flexible, resource-aware expert combinations. |
| Outcome: | The proposed model matches the performance of existing Mixture-of-Experts architectures while maintaining balanced GPU utilization. |
A Closer Look into Mixture-of-Experts in Large Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Mixture-of-experts (MoE) architectures are gaining increasing attention for their unique properties and remarkable performance. |
| Approach: | They propose a mixture-of-experts architecture that allows for model scaling without sacrificing computational efficiency. |
| Outcome: | The proposed model increases model size without sacrificing computational efficiency . the proposed model is modular and can be used by a broad spectrum of practitioners . |
HookMoE: A learnable performance compensation strategy of Mixture-of-Experts for LLM inference acceleration (2025.emnlp-main)
Copied to clipboard
| Challenge: | Mixture of Experts (MoE) models have been a promising paradigm for scaling model capacity through top-k routing mechanisms. |
| Approach: | They propose a plug-and-play single-layer compensation framework that strategically inserts a lightweight trainable Hook module immediately preceding selected transformer blocks. |
| Outcome: | The proposed framework reduces the number of activated experts by more than 50% and achieves a 1.42 inference speed-up during the prefill stage. |
HyperMoE: Towards Better Mixture of Experts via Transferring Among Experts (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods for enhancing performance through increased use of expert knowledge often result in diminishing sparsity during expert selection. |
| Approach: | They propose a framework that integrates the computational processes of MoE with the concept of knowledge transferring in multi-task learning. |
| Outcome: | The proposed framework outperforms existing methods under identical conditions concerning the number of experts. |
AEA: Adaptive Expert Allocation Improves Sentence Embeddings from Mixture-of-Experts LLM (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods to improve embeddings from Mixture-of-Experts models allocate a fixed number of experts uniformly across all layers and tokens, ignoring inter-layer and inter-token heterogeneity. |
| Approach: | They propose an Adaptive Expert Allocation framework that performs layer-wise and token-wise expert allocation to enhance embedding quality. |
| Outcome: | The proposed method improves embedding quality across multiple MoE models. |