MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decomposition

Challenge:	emergence of Mixture of Experts (MoE) LLMs has significantly advanced the development of language models.
Approach:	They propose a two-stage compression method tailored for Mixture of Experts to reduce the model size and decrease the computational cost.
Outcome:	The proposed method reduces model size and improves inference efficiency while maintaining performance in various zero-shot tasks.

Similar Papers

Overview

Challenge

Approach

Outcome

Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models (2024.acl-long)

Copied to clipboard

Xudong Lu, Qi Liu, Yuhui Xu, Aojun Zhou, Siyuan Huang, Bo Zhang, Junchi Yan, Hongsheng Li

Semantic Text Processing

Challenge:	Mixture-of-Experts (MoE) LLMs achieve higher performance with fewer active parameters, but are still difficult to deploy due to their immense parameter sizes.
Approach:	They propose expert-level sparsification techniques to enhance the deployment efficiency of large language models by introducing plug-and-play expert pruning and skipping techniques.
Outcome:	The proposed methods reduce model sizes and increase inference speed while maintaining satisfactory performance across a wide range of tasks.

Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts (2025.findings-acl)

Copied to clipboard

Zeliang Zhang, Xiaodong Liu, Hao Cheng, Chenliang Xu, Jianfeng Gao

Challenge:	Large Language Models (LLMs) have outstanding performance by learning a large number of model parameters on large amounts of data.
Approach:	They propose a method of grouping and pruning similar experts to improve the model’s parameter efficiency by a range of natural language tasks.
Outcome:	The proposed method outperforms other model pruning methods on a range of natural language tasks.

EAC-MoE: Expert-Selection Aware Compressor for Mixture-of-Experts Large Language Models (2025.acl-long)

Copied to clipboard

Yuanteng Chen, Yuantian Shao, Peisong Wang, Jian Cheng

Language Models

Challenge:	Mixture-of-Experts (MoE) has demonstrated promising potential in scaling LLMs . however, it is hindered by two critical challenges: substantial GPU memory consumption and low activated parameters.
Approach:	They propose an Expert-Selection Aware Compressor for Mixture-of-Experts (MoE) that aligns with the characteristics of MoE from the perspectives of quantization and pruning.
Outcome:	The proposed approach significantly reduces memory usage and improves inference speed with minimal performance degradation.

COMPEL: Compensated Mixture-of-Experts Pruning with Expert-Layer distribution (2026.findings-acl)

Copied to clipboard

Seohee Yoon, Yong Suk Choi

Language Models

Challenge:	Mixture-of-Experts (MoE) architectures are effective for scaling Large Language Models (LLMs) however, existing pruning methods adopt uniform pruning across layers, which fails to capture layer-wise variations in expert importance and redundancy.
Approach:	They propose a Mixture-of-Experts pruning method that activates only a subset of experts during inference by estimating expert importance using Fisher information.
Outcome:	The proposed pruning method outperforms existing pruning methods while reducing inference latency and peak GPU memory usage.

Efficiently Editing Mixture-of-Experts Models with Compressed Experts (2025.findings-emnlp)

Copied to clipboard

Yifei He, Yang Liu, Chen Liang, Hany Hassan Awadalla

Green & Sustainable NLP

Challenge:	Mixture-of-Experts models allow for efficient scaling of large language models . fewer experts reduce computational costs, while more experts improve performance .
Approach:	They propose to activate only a subset of experts during training and inference . they propose compressed experts that preserve the most important experts .
Outcome:	The proposed approach preserves the most important experts while replacing other auxiliary activated experts with compressed experts.

STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruning (2025.acl-long)

Copied to clipboard

Jaeseong Lee, Seung-won Hwang, Aurick Qiao, Daniel F Campos, Zhewei Yao, Yuxiong He

Language Models

Challenge:	Mixture-of-experts (MoEs) have been adopted for reducing inference costs by sparsely activating experts in large language models (LLMs).
Approach:	They propose a structured-then-unstructured approach outperforming both of structured and unstructured pruning for MoEs.
Outcome:	The proposed approach outperforms both of structured and unstructured pruning, especially for MoEs with hundreds of experts.

DIVE into MoE: Diversity-Enhanced Reconstruction of Large Language Models from Dense into Mixture-of-Experts (2025.acl-long)

Copied to clipboard

Yuchen Feng, Bowen Shen, Naibin Gu, Jiaxuan Zhao, Peng Fu, Zheng Lin, Weiping Wang

Semantic Text Processing

Challenge:	Existing methods for reconstruction of large language models overlook diversity among experts, leading to potential redundancy.
Approach:	They propose a pruning-based expert reconstruction method that prunes a specific LLM and retrains it on routers, experts and normalization modules.
Outcome:	The proposed method outperforms pruning and MoE reconstruction methods on Llama-style models with open-source training corpora.

Decoding Knowledge Attribution in Mixture-of-Experts: A Framework of Basic-Refinement Collaboration and Efficiency Analysis (2025.acl-long)

Copied to clipboard

Junzhuo Li, Bo Wang, Xiuze Zhou, Peijie Jiang, Jia Liu, Xuming Hu

Responsible & Trustworthy NLP

Challenge:	Existing attribution methods for dense models fail to capture dynamic routing-expert interactions in sparse MoE architectures.
Approach:	They propose to analyze sparse MoE architectures against dense models to capture dynamic routing-expert interactions.
Outcome:	The proposed algorithm shows that sparse models achieve higher efficiency per layer . it also shows that deep Qwen-MoE mitigates expert failures while minimizing complexity .

A Closer Look into Mixture-of-Experts in Large Language Models (2025.findings-naacl)

Copied to clipboard

Ka Man Lo, Zeyu Huang, Zihan Qiu, Zili Wang, Jie Fu

Language Models

Challenge:	Mixture-of-experts (MoE) architectures are gaining increasing attention for their unique properties and remarkable performance.
Approach:	They propose a mixture-of-experts architecture that allows for model scaling without sacrificing computational efficiency.
Outcome:	The proposed model increases model size without sacrificing computational efficiency . the proposed model is modular and can be used by a broad spectrum of practitioners .

Mixture of Heterogeneous Grouped Experts for Language Modeling (2026.acl-industry)

Copied to clipboard

Zhicheng Ma, Xiang Liu, Zhaoxiang Liu, Ning Wang, Yi Shen, Kai Wang, Shuming Shi, Shiguo Lian

Semantic Text Processing

Challenge:	Large Language Models (LLMs) based on Mixture-of-Experts (MoE) enforce uniform expert sizes, creating a rigidity that fails to align computational costs with varying token-level complexity.
Approach:	They propose a mixture of heterogeneous grouped experts (MoHGE) that allows for flexible, resource-aware expert combinations.
Outcome:	The proposed model matches the performance of existing Mixture-of-Experts architectures while maintaining balanced GPU utilization.