CoMoL: Efficient Mixture of LoRA Experts via Dynamic Core Space Merging

Challenge:	Existing PEFT methods suffer from limited parameter efficiency and coarse-grained adaptation due to proliferation of LoRA experts and instance-level routing.
Approach:	They propose a new MoE-LoRA framework that incorporates expert diversity, parameter efficiency, and fine-grained adaptation.
Outcome:	The proposed framework outperforms existing methods on multiple tasks while maintaining parameter efficiency.

Similar Papers

Overview

Challenge

Approach

Outcome

MoA: Heterogeneous Mixture of Adapters for Parameter-Efficient Fine-Tuning of Large Language Models (2026.acl-long)

Copied to clipboard

Jie Cao, Tianwei Lin, Bo Yuan, Rolan Yan, Hongyang He, Wenqiao Zhang, Juncheng Li, Dongping Zhang, Siliang Tang, Yueting Zhuang

Semantic Text Processing

Challenge:	Existing methods for parameter-efficient fine-tuning (PEFT) are limited by computational costs and performance degradation.
Approach:	They propose a method that integrates Low-Rank Adaptation and Mixture-of-Experts (MoE) they propose combining expert load imbalance and representation collapse to improve LLM performance .
Outcome:	The proposed method outperforms homogeneous MoE-LoRA architectures in performance and parameter efficiency.

LoRACoE: Improving Large Language Model via Composition-based LoRA Expert (2025.emnlp-main)

Copied to clipboard

Guanyu Li, Zhiheng Xi, Zhihao Zhang, Boyang Hong, Tao Gui, Qi Zhang, Xuanjing Huang

Language Models

Challenge:	Recent studies show that the Mixture of Experts architecture improves performance of large language models.
Approach:	They propose a method to build static experts using LoRA parameters . they propose to use rank-level parameters to build experts based on rank-based parameters based in LoRA module.
Outcome:	The proposed method improves task performance across a broader range of tasks.

SLIM: Let LLM Learn More and Forget Less with Soft LoRA and Identity Mixture (2025.naacl-long)

Copied to clipboard

Jiayi Han, Liang Du, Hongwei Du, Xiangguo Zhou, Yiwen Wu, Yuanfang Zhang, Weibo Zheng, Donghong Han

Green & Sustainable NLP

Challenge:	balancing the training budget, downstream performance, and general capabilities of large language models remains a challenge in many applications.
Approach:	They propose a mixture of expert framework based on Soft LoRA and Identity Mixture . SLIM allows dynamic routing between LoRA adapters and identity layers .
Outcome:	The proposed framework reduces training cost while maintaining general capabilities . it can be open-sourced upon publication.

MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuning (2024.findings-emnlp)

Copied to clipboard

Jingfan Zhang, Yi Zhao, Dan Chen, Xing Tian, Huanran Zheng, Wei Zhu

Language Models

Challenge:	Low-rank adaptation and its mixture-of-experts (MOE) methods are highly effective but introduce significant latency in multi-tenant settings due to the LoRA modules and MOE routers added to multiple linear modules.
Approach:	They propose a low-rank adaptation variant that considers each LoRA module as an expert and employs a prompt-aware routing mechanism.
Outcome:	Extensive analysis on commonsense reasoning tasks and math reasoning tasks show that MiLoRA outperforms strong PEFT baselines with comparable tunable parameter budgets.

Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Models (2024.emnlp-main)

Copied to clipboard

Zihan Wang, Deli Chen, Damai Dai, Runxin Xu, Zhuoshu Li, Yu Wu

Semantic Text Processing

Challenge:	Existing studies on parameter-efficient fine-tuning (PEFT) for dense-architecture LLMs are lacking.
Approach:	They propose an expert-specialized fine-tuning method that tunes the experts most relevant to downstream tasks while freezing the other experts.
Outcome:	The proposed method matches or surpasses full-parameter fine-tuning.

Parameter-Efficient Routed Fine-Tuning: Mixture-of-Experts Demands Mixture of Adaptation Modules (2026.findings-eacl)

Copied to clipboard

Yilun Liu, Yunpu Ma, Yuetian Lu, Shuo Chen, Zifeng Ding, Volker Tresp

Language Models

Challenge:	Existing Parameter-Efficient Fine-Tuning (PEFT) strategies that focus on specialized experts are not effective for Mixture-of-Experts (MoE).
Approach:	They propose to integrate a dynamic routing mechanism among specialized experts in Mixture-of-Experts (MoE) .
Outcome:	Extensive experiments on commonsense and math reasoning tasks validate the performance and efficiency of the proposed routed approach.

SAMoRA: Semantic-Aware Mixture of LoRA Experts for Task-Adaptive Learning (2026.findings-acl)

Copied to clipboard

Boyan Shi, Wei Chen, Shuyuan Zhao, Junfeng Shen, Shengnan Guo, Shaojiang Wang, Huaiyu Wan

Language Models

Challenge:	Existing methods for multitask learning fail to match input semantics with expert capabilities, leading to weak expert specialization.
Approach:	They propose a parameter-efficient mixture-of-experts framework for task-adaptive learning that aligns textual semantics with the most suitable experts for precise routing.
Outcome:	The proposed framework outperforms the state-of-the-art methods and holds excellent task generalization capabilities.

GuiLoMo: Allocating Experts and Ranks for LoRA-MoE via Bilevel Optimization with GuidedSelection Vectors (2025.findings-emnlp)

Copied to clipboard

Xinrong Chen, Hengyuan Zhang, Yingmin Qiu, Xiao Liang, Ziyue Li, Guanyu Wang, Weiping Li, Tong Mo, Hayden Kwok-Hay So, Ngai Wong

Green & Sustainable NLP

Challenge:	Low-Rank Adaptation (LoRA) methods are efficient for a large language model with reduced computational costs.
Approach:	They propose a layer-wise expert numbers and ranks allocation strategy with GuidedSelection Vectors.
Outcome:	The proposed method achieves superior or comparable performance to all baselines on three backbone models.

MoDE: Effective Multi-task Parameter Efficient Fine-Tuning with a Mixture of Dyadic Experts (2025.findings-naacl)

Copied to clipboard

Lin Ning, Harsh Lara, Meiqi Guo, Abhinav Rastogi

Responsible & Trustworthy NLP

Challenge:	Recent efforts have explored mixtures of LoRA modules for multi-task settings, but this study reveals redundancy in the down-projection matrix of these architectures.
Approach:	They propose a method to share down-projection matrix across tasks and employ atomic rank-one adapters coupled with routers that allow more sophisticated task-level specialization.
Outcome:	The proposed method outperforms state-of-the-art models on a SNI benchmark and provides a practical solution for deploying lightweight models.

A Closer Look into Mixture-of-Experts in Large Language Models (2025.findings-naacl)

Copied to clipboard

Ka Man Lo, Zeyu Huang, Zihan Qiu, Zili Wang, Jie Fu

Language Models

Challenge:	Mixture-of-experts (MoE) architectures are gaining increasing attention for their unique properties and remarkable performance.
Approach:	They propose a mixture-of-experts architecture that allows for model scaling without sacrificing computational efficiency.
Outcome:	The proposed model increases model size without sacrificing computational efficiency . the proposed model is modular and can be used by a broad spectrum of practitioners .