| Challenge: | Expert pruning is a practical deployment technique for Mixture-of-Experts models . but its success depends heavily on the calibration set used for pruning . |
| Approach: | They propose a calibration tool that compares expert activations across datasets to predict calibration perturbations without running expensive pruning procedures. |
| Outcome: | The proposed system compares expert activations across datasets to predict calibration perturbations without running expensive pruning procedures. |
Similar Papers
COMPEL: Compensated Mixture-of-Experts Pruning with Expert-Layer distribution (2026.findings-acl)
Copied to clipboard
| Challenge: | Mixture-of-Experts (MoE) architectures are effective for scaling Large Language Models (LLMs) however, existing pruning methods adopt uniform pruning across layers, which fails to capture layer-wise variations in expert importance and redundancy. |
| Approach: | They propose a Mixture-of-Experts pruning method that activates only a subset of experts during inference by estimating expert importance using Fisher information. |
| Outcome: | The proposed pruning method outperforms existing pruning methods while reducing inference latency and peak GPU memory usage. |
Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have outstanding performance by learning a large number of model parameters on large amounts of data. |
| Approach: | They propose a method of grouping and pruning similar experts to improve the model’s parameter efficiency by a range of natural language tasks. |
| Outcome: | The proposed method outperforms other model pruning methods on a range of natural language tasks. |
Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Mixture-of-Experts (MoE) LLMs achieve higher performance with fewer active parameters, but are still difficult to deploy due to their immense parameter sizes. |
| Approach: | They propose expert-level sparsification techniques to enhance the deployment efficiency of large language models by introducing plug-and-play expert pruning and skipping techniques. |
| Outcome: | The proposed methods reduce model sizes and increase inference speed while maintaining satisfactory performance across a wide range of tasks. |
STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruning (2025.acl-long)
Copied to clipboard
| Challenge: | Mixture-of-experts (MoEs) have been adopted for reducing inference costs by sparsely activating experts in large language models (LLMs). |
| Approach: | They propose a structured-then-unstructured approach outperforming both of structured and unstructured pruning for MoEs. |
| Outcome: | The proposed approach outperforms both of structured and unstructured pruning, especially for MoEs with hundreds of experts. |
MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decomposition (2024.findings-emnlp)
Copied to clipboard
Cheng Yang, Yang Sui, Jinqi Xiao, Lingyi Huang, Yu Gong, Yuanlin Duan, Wenqi Jia, Miao Yin, Yu Cheng, Bo Yuan
| Challenge: | emergence of Mixture of Experts (MoE) LLMs has significantly advanced the development of language models. |
| Approach: | They propose a two-stage compression method tailored for Mixture of Experts to reduce the model size and decrease the computational cost. |
| Outcome: | The proposed method reduces model size and improves inference efficiency while maintaining performance in various zero-shot tasks. |
LightMoE: Task-Aware Expert Availability Management for Memory-Efficient MoE-LLM Inference (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing solutions for balancing model accuracy with inference latency are limited due to memory constraints. |
| Approach: | They propose a framework for memory-efficient MoE inference that exploits the functional redundancy and temporal locality of expert activation. |
| Outcome: | The proposed framework improves accuracy-efficiency trade-off by 4.3% over pruned models and 2.4% over dynamic swapping methods while maintaining inference latency comparable to pruned model. |
Decoding Knowledge Attribution in Mixture-of-Experts: A Framework of Basic-Refinement Collaboration and Efficiency Analysis (2025.acl-long)
Copied to clipboard
| Challenge: | Existing attribution methods for dense models fail to capture dynamic routing-expert interactions in sparse MoE architectures. |
| Approach: | They propose to analyze sparse MoE architectures against dense models to capture dynamic routing-expert interactions. |
| Outcome: | The proposed algorithm shows that sparse models achieve higher efficiency per layer . it also shows that deep Qwen-MoE mitigates expert failures while minimizing complexity . |
EAC-MoE: Expert-Selection Aware Compressor for Mixture-of-Experts Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Mixture-of-Experts (MoE) has demonstrated promising potential in scaling LLMs . however, it is hindered by two critical challenges: substantial GPU memory consumption and low activated parameters. |
| Approach: | They propose an Expert-Selection Aware Compressor for Mixture-of-Experts (MoE) that aligns with the characteristics of MoE from the perspectives of quantization and pruning. |
| Outcome: | The proposed approach significantly reduces memory usage and improves inference speed with minimal performance degradation. |
Efficiently Editing Mixture-of-Experts Models with Compressed Experts (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Mixture-of-Experts models allow for efficient scaling of large language models . fewer experts reduce computational costs, while more experts improve performance . |
| Approach: | They propose to activate only a subset of experts during training and inference . they propose compressed experts that preserve the most important experts . |
| Outcome: | The proposed approach preserves the most important experts while replacing other auxiliary activated experts with compressed experts. |
Towards Robust Pruning: An Adaptive Knowledge-Retention Pruning Strategy for Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing pruning strategies struggle to enhance robustness against adversarial attacks when continually increasing model sparsity and require a retraining process. |
| Approach: | They propose a pruning strategy that replicates embedding space and feature space of dense language models and aims to conserve more pre-trained knowledge during the pruning process. |
| Outcome: | The proposed pruning strategy replicates embedding space and feature space of dense language models, aiming to conserve more pre-trained knowledge during the pruning process. |