Exploration-Driven Reinforcement Learning for Expert Routing Improvement in Mixture-of-Experts Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | MoE-based LLMs are not explicitly supervised to select suitable experts. |
| Approach: | They propose Exploration-Driven Reinforcement Learning (ERL) which explicitly optimizes the router by exploration of alternative routing paths. |
| Outcome: | The proposed method improves summarization (SAMSum, XSUM, question answering, and language modeling), and raises routing quality, delivering 8.9 higher MRR than baselines over 100 perturbed routing paths. |
Similar Papers
A Closer Look into Mixture-of-Experts in Large Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Mixture-of-experts (MoE) architectures are gaining increasing attention for their unique properties and remarkable performance. |
| Approach: | They propose a mixture-of-experts architecture that allows for model scaling without sacrificing computational efficiency. |
| Outcome: | The proposed model increases model size without sacrificing computational efficiency . the proposed model is modular and can be used by a broad spectrum of practitioners . |
Improved Policy Optimization for Mixture-of-Experts Models: Importance Sampling and Rewarding from an Expert-Centric Perspective (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to reinforcement learning (RL) suffer from training instability . existing approaches often ignore token-specific discrepancies in expert assignments . |
| Approach: | They propose to introduce expert-level importance sampling to reduce complexity of RL . they propose to leverage expert-centric granularity to ensure a rigorous alignment between reward signals and policy updates. |
| Outcome: | The proposed method outperforms strong baselines across reasoning tasks. |
Probing Semantic Routing in Large Mixture-of-Expert Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | large mixture-of-expert models have become increasingly common in the open domain . prior work has explored functional differentiation through routing behavior . |
| Approach: | They investigate whether expert routing in large mixture-of-expert models is influenced by the semantics of the inputs. |
| Outcome: | The results show that expert routing is influenced by the semantics of the inputs. |
Harder Task Needs More Experts: Dynamic Routing in MoE Models (2024.acl-long)
Copied to clipboard
Quzhe Huang, Zhenwei An, Nan Zhuang, Mingxu Tao, Chen Zhang, Yang Jin, Kun Xu, Kun Xu, Liwei Chen, Songfang Huang, Yansong Feng
| Challenge: | Unlike existing MoE approaches that rely on fixed TopK Routing, our dynamic expert selection framework dynamically allocates experts based on the confidence level in expert selection for each input. |
| Approach: | They propose a dynamic expert selection framework that dynamically allocates experts based on the confidence level in expert selection for each input. |
| Outcome: | The proposed method achieves an average improvement of 0.7% with less than 90% activated parameters and outperforms dense models in QA and machine translation tasks. |
On the Benefits of Learning to Route in Mixture-of-Experts Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing Mixture-of-Expert (MoE) models allow us to scale up model sizes while keeping the amount of compute time fixed. |
| Approach: | They propose to use a router to route inputs to experts in a layer to scale up model sizes while keeping the amount of compute time fixed. |
| Outcome: | The proposed model scales up with the help of a router that routes input tokens to experts in a layer and shows that it is more efficient than a non-trainable router. |
Enhancing Efficiency and Exploration in Reinforcement Learning for LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches allocate an equal number of rollouts to all questions during the RL process, which is inefficient. |
| Approach: | They propose a mechanism for dynamically allocating rollout budgets based on the difficulty of the problems, enabling more efficient RL training. |
| Outcome: | The proposed model improves response precision while preserving exploratory ability to uncover potential correct pathways. |
Parameter-Efficient Routed Fine-Tuning: Mixture-of-Experts Demands Mixture of Adaptation Modules (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing Parameter-Efficient Fine-Tuning (PEFT) strategies that focus on specialized experts are not effective for Mixture-of-Experts (MoE). |
| Approach: | They propose to integrate a dynamic routing mechanism among specialized experts in Mixture-of-Experts (MoE) . |
| Outcome: | Extensive experiments on commonsense and math reasoning tasks validate the performance and efficiency of the proposed routed approach. |
Part-Of-Speech Sensitivity of Routers in Mixture of Experts Models (2025.coling-main)
Copied to clipboard
| Challenge: | a study examines the behavior of routers in Mixture of Experts (MoE) models . experts with similar linguistic traits are often routed to the same expert regardless of context . |
| Approach: | They investigate how tokens are routed based on their linguistic features . they aim to explore whether experts specialize in processing tokens with similar linguistic traits . |
| Outcome: | The proposed model-integrated routers are based on Mixture of Experts (MoE) models . the results show that expert specialization is high for POS categories . |
Exploring Domain Robust Lightweight Reward Models based on Router Mechanism (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in large language models have relied on the large reward model for fine-tuning, but the use of a single reward model across domains may not always be optimal. |
| Approach: | They propose to use router mechanisms to train small language models in a domain-specific manner . they use internal routers, external routers and router adapters to create a single reward model . |
| Outcome: | The proposed approach reduces parameter size while minimizing parameter size. |
Towards Stable and Effective Reinforcement Learning for Mixture-of-Experts (2026.acl-long)
Copied to clipboard
| Challenge: | Reinforcement learning with verifiable rewards (RLVR) training with Mixture-of-Experts policies remains fragile and prone to reward collapse. |
| Approach: | They propose a router shift-based policy optimization method that computes a per-token router-shift ratio conditioned on the previously activated experts and applies stop-gradient and a lower-bound floor. |
| Outcome: | The proposed method achieves better performance and greater stability than previous methods. |