Papers by Daiyi Peng
MoLA: MoE LoRA with Layer-wise Expert Allocation (2025.findings-naacl)
Copied to clipboard
Chongyang Gao, Kezhen Chen, Jinmeng Rao, Ruibo Liu, Baochen Sun, Yawen Zhang, Daiyi Peng, Xiaoyuan Guo, Vs Subrahmanian
| Challenge: | Recent efforts to integrate low-rank adaptation (LoRA) with the Mixture-of-Experts (MoE) have achieved performance comparable to full-parameter fine-tuning by tuning much fewer parameters. |
| Approach: | They propose a parameter-efficient MoE method for low-rank adaptation with the Mixture-of-Experts (MoE) they use layers of LoRA experts to allocate more LoRA expert to middle layers . |
| Outcome: | The proposed method outperforms baseline models on six well-known NLP and commonsense QA benchmarks on LLAMA-2, Mistral, and Gemma. |