CartesianMoE: Boosting Knowledge Sharing among Experts via Cartesian Product Routing in Mixture-of-Experts

Challenge:	Large language models (LLMs) have been attracting much attention due to their impressive performance in all kinds of downstream tasks.
Approach:	They propose a mix-of-experts model that allows the model size to grow without raising training costs.
Outcome:	The proposed model outperforms existing models in perplexity and robustness tests.

Similar Papers

Overview

Challenge

Approach

Outcome

HyperMoE: Towards Better Mixture of Experts via Transferring Among Experts (2024.acl-long)

Copied to clipboard

Hao Zhao, Zihan Qiu, Huijia Wu, Zili Wang, Zhaofeng He, Jie Fu

Language Models

Challenge:	Existing methods for enhancing performance through increased use of expert knowledge often result in diminishing sparsity during expert selection.
Approach:	They propose a framework that integrates the computational processes of MoE with the concept of knowledge transferring in multi-task learning.
Outcome:	The proposed framework outperforms existing methods under identical conditions concerning the number of experts.

A Closer Look into Mixture-of-Experts in Large Language Models (2025.findings-naacl)

Copied to clipboard

Ka Man Lo, Zeyu Huang, Zihan Qiu, Zili Wang, Jie Fu

Language Models

Challenge:	Mixture-of-experts (MoE) architectures are gaining increasing attention for their unique properties and remarkable performance.
Approach:	They propose a mixture-of-experts architecture that allows for model scaling without sacrificing computational efficiency.
Outcome:	The proposed model increases model size without sacrificing computational efficiency . the proposed model is modular and can be used by a broad spectrum of practitioners .

Probing Semantic Routing in Large Mixture-of-Expert Models (2025.findings-emnlp)

Copied to clipboard

Matthew Lyle Olson, Neale Ratzlaff, Musashi Hinck, Man Luo, Sungduk Yu, Chendi Xue, Vasudev Lal

Responsible & Trustworthy NLP

Challenge:	large mixture-of-expert models have become increasingly common in the open domain . prior work has explored functional differentiation through routing behavior .
Approach:	They investigate whether expert routing in large mixture-of-expert models is influenced by the semantics of the inputs.
Outcome:	The results show that expert routing is influenced by the semantics of the inputs.

Mixture of Heterogeneous Grouped Experts for Language Modeling (2026.acl-industry)

Copied to clipboard

Zhicheng Ma, Xiang Liu, Zhaoxiang Liu, Ning Wang, Yi Shen, Kai Wang, Shuming Shi, Shiguo Lian

Semantic Text Processing

Challenge:	Large Language Models (LLMs) based on Mixture-of-Experts (MoE) enforce uniform expert sizes, creating a rigidity that fails to align computational costs with varying token-level complexity.
Approach:	They propose a mixture of heterogeneous grouped experts (MoHGE) that allows for flexible, resource-aware expert combinations.
Outcome:	The proposed model matches the performance of existing Mixture-of-Experts architectures while maintaining balanced GPU utilization.

Specialization through Collaboration: Understanding Expert Interaction in Mixture-of-Expert Large Language Models (2026.eacl-long)

Copied to clipboard

Yuanbo Tang, Naifan Zhang, Yan Tang, Meixuan Chen, Shuhan Huang, Tingyu Cao, Yang Li

Language Models

Challenge:	Mixture-of-Experts (MoE) based large language models are popular for multitasking . however, whether each expert can specialize to a task remains unclear .
Approach:	They propose to use a dictionary learning approach to analyze expert collaboration mechanisms in MoE LLMs.
Outcome:	The proposed model outperforms existing methods by 2.5% while enabling 50% expert reduction.

ModularMoE: Fast LLM Customization with Parameter-Sharing Mixture-of-Experts for Low-Resource Settings (2026.findings-acl)

Copied to clipboard

Jiaxing Liu, Qi Qi, Haifeng Sun, Dunjun Li, Zirui Zhuang, Bo He, Xiang Yang, Cong Liu, Jianxin Liao, Jingyu Wang

Responsible & Trustworthy NLP

Challenge:	Large Language Models impose significant computational and storage burdens on personal devices . existing customization approaches incur excessive computational costs or lead to suboptimal performance .
Approach:	They propose a training framework that converts pre-trained LLMs into parameter-sharing MoE models for lightweight deployment.
Outcome:	The proposed training framework outperforms state-of-the-art training frameworks at the same sparsity level while delivering up to 2.71 inference speedup.

From Pseudo-Balancing to True Specialization: Memory-Aware Routing for Mixture-of-Experts (2026.findings-acl)

Copied to clipboard

Peixuan Hou, Yunbo Hou, Bin Chen, LI He, Jian Xu, Weiping Li, Bo Zheng, Guojie Song

Green & Sustainable NLP

Challenge:	Existing methods to optimize expert-centered load balancing fail to account for pseudo-balance phenomenon . severe knowledge overlap among experts leads to redundant representations and inefficient parameter utilization .
Approach:	They propose a method that prioritizes expert utilization over semantic alignment . they use memory-aware routing to ensure expert load balancing is consistent .
Outcome:	Experimental results show that MAR improves expert specialization by 35% and accuracy by 2%-25% . MAR matches baseline performance with only half the experts .

Advancing MoE Efficiency: A Collaboration-Constrained Routing (C2R) Strategy for Better Expert Parallelism Design (2025.naacl-long)

Copied to clipboard

Mohan Zhang, Pingzhi Li, Jie Peng, Mufan Qiu, Tianlong Chen

Green & Sustainable NLP

Challenge:	Using Mixture-of-Experts, researchers have found that efficient MoE is difficult to achieve due to two key reasons: imbalanced expert activation and massive communication overhead.
Approach:	They propose a collaboration-constrained routing strategy that encourages more specialized expert groups and leverages expert specialization.
Outcome:	The proposed approach achieves an average performance improvement of 0.51% and 0.33% on LLaMA-MoE and Qwen-MaE respectively.

StableMoE: Stable Routing Strategy for Mixture of Experts (2022.acl-long)

Copied to clipboard

Damai Dai, Li Dong, Shuming Ma, Bo Zheng, Zhifang Sui, Baobao Chang, Furu Wei

Challenge:	Existing learning-to-route methods suffer from the routing fluctuation issue . with the model scale growing, training speed will go slower and memory requirements are heavy .
Approach:	They propose a Mixture-of-Experts technique that can scale up the model size of Transformers with an affordable computational overhead.
Outcome:	The proposed method outperforms existing learning-to-route methods on language modeling and multilingual machine translation.

Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference (2021.findings-emnlp)

Copied to clipboard

Sneha Kudugunta, Yanping Huang, Ankur Bapna, Maxim Krikun, Dmitry Lepikhin, Minh-Thang Luong, Orhan Firat

Green & Sustainable NLP

Challenge:	Sparse Mixture-of-Experts (MoE) is a successful approach for scaling multilingual translation models to billions of parameters without a proportional increase in training computation.
Approach:	They propose to use a task-level routing approach to extract smaller, ready-to-deploy sub-networks from large sparse models by ignoring distillation.
Outcome:	Experiments on WMT and a web-scale dataset show that task-level routing outperforms token-level MoE models by +1.0 BLEU on average across 30 language pairs.