Nexus: Adaptive Upcycling to Efficiently Pretrain Mixture of Experts (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Nevertheless, training from scratch on trillions of tokens remains expensive that most users can only finetune these models. |
| Approach: | They propose to reuse parameters of dense models for the MoE layers with a router that can integrate new experts into existing trained models without hurting performance on previous domains. |
| Outcome: | The proposed router can integrate new experts into existing trained models without hurting the performance on previous domains. |
Similar Papers
Upcycling Instruction Tuning from Dense to Mixture-of-Experts via Parameter Merging (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for tuning large language models from dense to MoE face significant data requirements and require large-scale post-training. |
| Approach: | They propose an upcycling instruction tuning approach for tuning a dense pre-trained model into a MoE instruction model using genetic algorithm and parameter merging. |
| Outcome: | The proposed approach improves the performance of large language models with a small amount of seed data and improves their scaling. |
A Closer Look into Mixture-of-Experts in Large Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Mixture-of-experts (MoE) architectures are gaining increasing attention for their unique properties and remarkable performance. |
| Approach: | They propose a mixture-of-experts architecture that allows for model scaling without sacrificing computational efficiency. |
| Outcome: | The proposed model increases model size without sacrificing computational efficiency . the proposed model is modular and can be used by a broad spectrum of practitioners . |
Improved Sparse Upcycling for Instruction Tuning (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods for sparse upcycling lead to performance degradation in instruction tuning scenarios. |
| Approach: | They propose a representation-based approach to convert dense language models into sparsely activated ones by initializing router weights from language models. |
| Outcome: | The proposed architecture improves model capabilities and routing consistency across multiple benchmarks. |
HyperRouter: Towards Efficient Training and Inference of Sparse Mixture of Experts (2023.emnlp-main)
Copied to clipboard
Truong Do, Le Khiem, Quang Pham, TrungTin Nguyen, Thanh-Nam Doan, Binh Nguyen, Chenghao Liu, Savitha Ramasamy, Xiaoli Li, Steven Hoi
| Challenge: | Recent studies suggest that fixing the routers can achieve competitive performance by alleviating the collapsing problem, where all experts eventually learn similar representations. |
| Approach: | They propose a method that dynamically generates router parameters through a fixed hypernetwork and trainable embeddings to achieve a balance between training the routers and freezing them to learn an improved routing policy. |
| Outcome: | Experiments on a wide range of tasks show that the proposed method performs better than existing methods. |
Parameter-Efficient Routed Fine-Tuning: Mixture-of-Experts Demands Mixture of Adaptation Modules (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing Parameter-Efficient Fine-Tuning (PEFT) strategies that focus on specialized experts are not effective for Mixture-of-Experts (MoE). |
| Approach: | They propose to integrate a dynamic routing mechanism among specialized experts in Mixture-of-Experts (MoE) . |
| Outcome: | Extensive experiments on commonsense and math reasoning tasks validate the performance and efficiency of the proposed routed approach. |
LoRACoE: Improving Large Language Model via Composition-based LoRA Expert (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show that the Mixture of Experts architecture improves performance of large language models. |
| Approach: | They propose a method to build static experts using LoRA parameters . they propose to use rank-level parameters to build experts based on rank-based parameters based in LoRA module. |
| Outcome: | The proposed method improves task performance across a broader range of tasks. |
Union-of-Experts: Neurons in Mixture-of-Experts are Secretly Routers (2026.acl-long)
Copied to clipboard
| Challenge: | Mixture-of-Experts (MoE) models rely on an external router to assign tokens to experts, resulting in suboptimal performance. |
| Approach: | They propose an MoE variant that performs "expert-autonomous routing" by pre-designating a fraction of neurons within each expert as "routing neurons" they pre-train UoE models with up to 3B parameters and show they outperform traditional MoEs with matched efficiency. |
| Outcome: | The proposed model outperforms existing models with 3B parameters and provides valuable insights into expert-autonomous selection and the broader routing mechanisms of MoE models. |
MixtureKit: A General Framework for Composing, Training, and Visualizing Mixture-of-Experts Models (2026.acl-demo)
Copied to clipboard
| Challenge: | MixtureKit is a modular open-source framework for constructing, training, and analyzing Mixture-of-Experts (MoE) models from arbitrary pre-trained or fine-tuned checkpoints. |
| Approach: | They propose a modular open-source framework for constructing, training, and analyzing Mixture-of-Experts (MoE) models from arbitrary pre-trained or fine-tuned checkpoints. |
| Outcome: | Experiments on multilingual code-switched (Arabic–Latin) show that BTX models built with MixtureKit outperform dense baselines across multiple benchmarks. |
Mixture of Diverse Size Experts (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Recent large language models (LLMs) have shown superior performance in a variety of tasks due to the sub-linearly increasing computational costs. |
| Approach: | They propose a new MoE architecture with designed layers where experts have different sizes to mitigate this defect. |
| Outcome: | The proposed architecture surpasses existing MoEs by adaptively assigning the parameter budget to experts while maintaining the same total parameter size and number of experts. |
SAMoRA: Semantic-Aware Mixture of LoRA Experts for Task-Adaptive Learning (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for multitask learning fail to match input semantics with expert capabilities, leading to weak expert specialization. |
| Approach: | They propose a parameter-efficient mixture-of-experts framework for task-adaptive learning that aligns textual semantics with the most suitable experts for precise routing. |
| Outcome: | The proposed framework outperforms the state-of-the-art methods and holds excellent task generalization capabilities. |