MoRE: A Mixture of Low-Rank Experts for Adaptive Multi-Task Learning (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have revolutionized various domains, offering unprecedented performance across numerous tasks. |
| Approach: | They propose a new Mixture of Low-Rank Experts (MoRE) for multi-task PEFT to improve performance of LLMs with fewer parameters. |
| Outcome: | The proposed method improves performance over multiple tasks and no additional inference cost. |
Similar Papers
CoLA: Collaborative Low-Rank Adaptation (2025.findings-acl)
Copied to clipboard
| Challenge: | The scaling law of Large Language Models (LLMs) reveals diminishing return on performance as model scale increases. |
| Approach: | They propose a more flexible LoRA architecture with an efficient initialization scheme . they propose combining three collaborative strategies to enhance performance . |
| Outcome: | The proposed model outperforms existing methods in low-sample scenarios. |
MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Low-rank adaptation and its mixture-of-experts (MOE) methods are highly effective but introduce significant latency in multi-tenant settings due to the LoRA modules and MOE routers added to multiple linear modules. |
| Approach: | They propose a low-rank adaptation variant that considers each LoRA module as an expert and employs a prompt-aware routing mechanism. |
| Outcome: | Extensive analysis on commonsense reasoning tasks and math reasoning tasks show that MiLoRA outperforms strong PEFT baselines with comparable tunable parameter budgets. |
MoDE: Effective Multi-task Parameter Efficient Fine-Tuning with a Mixture of Dyadic Experts (2025.findings-naacl)
Copied to clipboard
| Challenge: | Recent efforts have explored mixtures of LoRA modules for multi-task settings, but this study reveals redundancy in the down-projection matrix of these architectures. |
| Approach: | They propose a method to share down-projection matrix across tasks and employ atomic rank-one adapters coupled with routers that allow more sophisticated task-level specialization. |
| Outcome: | The proposed method outperforms state-of-the-art models on a SNI benchmark and provides a practical solution for deploying lightweight models. |
Sparsity May Be All You Need: Sparse Random Parameter Adaptation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Parameter-Efficient Fine-Tuning (PEFT) methods aim at reducing computational and memory resources for fine-tuning large language models. |
| Approach: | They propose to train on a small number of parameters instead of all model parameters . they compare the method to LoRA and find it to be efficient . |
| Outcome: | The proposed method is competitive with LoRA when using a similar number of trainable parameters. |
Beyond Full Fine-tuning: Harnessing the Power of LoRA for Multi-Task Instruction Tuning (2024.lrec-main)
Copied to clipboard
Chunlei Xin, Yaojie Lu, Hongyu Lin, Shuheng Zhou, Huijia Zhu, Weiqiang Wang, Zhongyi Liu, Xianpei Han, Le Sun
| Challenge: | Low-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning algorithm for large-scale language models. |
| Approach: | They conduct a systematic study of Low-Rank Adaptation (LoRA) on diverse tasks and rich resources with different learning capacities. |
| Outcome: | The proposed algorithm can achieve remarkable performance in high-resource and multi-task scenarios, even comparable to full fine-tuning. |
MALoRA: Mixture of Asymmetric Low-Rank Adaptation for Enhanced Multi-Task Learning (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) can be fine-tuned to new tasks, but in multi-task scenarios, training imbalance and seesaw effect often arise. |
| Approach: | They propose a flexible fine-tuning framework that leverages asymmetric optimization among LoRA experts to reduce training imbalance and improve performance. |
| Outcome: | The proposed framework outperforms baseline methods in inter- and intra-task learning scenarios. |
LoRA Soups: Merging LoRAs for Practical Skill Composition Tasks (2025.coling-industry)
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) is a popular technique for parameter-efficient fine-tuning of Large Language Models. |
| Approach: | They propose to combine LoRA modules to achieve skill composition . they propose to use concatenation of LoRAs to optimize weights for different LoRA training . |
| Outcome: | The proposed model outperforms existing models and data- merging techniques on math-word problems and domain-specialized corpora. |
GuiLoMo: Allocating Experts and Ranks for LoRA-MoE via Bilevel Optimization with GuidedSelection Vectors (2025.findings-emnlp)
Copied to clipboard
Xinrong Chen, Hengyuan Zhang, Yingmin Qiu, Xiao Liang, Ziyue Li, Guanyu Wang, Weiping Li, Tong Mo, Hayden Kwok-Hay So, Ngai Wong
| Challenge: | Low-Rank Adaptation (LoRA) methods are efficient for a large language model with reduced computational costs. |
| Approach: | They propose a layer-wise expert numbers and ranks allocation strategy with GuidedSelection Vectors. |
| Outcome: | The proposed method achieves superior or comparable performance to all baselines on three backbone models. |
MoA: Heterogeneous Mixture of Adapters for Parameter-Efficient Fine-Tuning of Large Language Models (2026.acl-long)
Copied to clipboard
Jie Cao, Tianwei Lin, Bo Yuan, Rolan Yan, Hongyang He, Wenqiao Zhang, Juncheng Li, Dongping Zhang, Siliang Tang, Yueting Zhuang
| Challenge: | Existing methods for parameter-efficient fine-tuning (PEFT) are limited by computational costs and performance degradation. |
| Approach: | They propose a method that integrates Low-Rank Adaptation and Mixture-of-Experts (MoE) they propose combining expert load imbalance and representation collapse to improve LLM performance . |
| Outcome: | The proposed method outperforms homogeneous MoE-LoRA architectures in performance and parameter efficiency. |
LoRACoE: Improving Large Language Model via Composition-based LoRA Expert (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show that the Mixture of Experts architecture improves performance of large language models. |
| Approach: | They propose a method to build static experts using LoRA parameters . they propose to use rank-level parameters to build experts based on rank-based parameters based in LoRA module. |
| Outcome: | The proposed method improves task performance across a broader range of tasks. |