LoRA-PAR: A Flexible Dual-System LoRA Partitioning Approach to Efficient LLM Fine-Tuning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large-scale generative models like DeepSeek-R1 and OpenAI-O1 benefit substantially from chain-of-thought reasoning, yet pushing their performance typically requires vast data, large model sizes, and full-parameter fine-tuning. |
| Approach: | They propose a dual-system LoRA framework that partitions data and parameters by System 1 or System 2 demands and adopts a two-stage fine-tuning strategy to enhance knowledge and intuition. |
| Outcome: | The proposed framework partitions data and parameters by System 1 or System 2 demands, using fewer yet more focused parameters for each task. |
Similar Papers
Rethinking Parameter Sharing for LLM Fine-Tuning with Multiple LoRAs (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for large language models use parameter-efficient techniques such as Low-Rank Adaptation (LoRA) prior studies suggest that the inner A matrices are highly similar during training and therefore suitable for sharing. |
| Approach: | They propose an asymmetric multi-LoRA design with multiple A matrices and a single shared B in multi-task fine-tuning. |
| Outcome: | The proposed methods achieve more balanced performance across tasks with comparable or superior average accuracy relative to existing methods. |
MoDE: Effective Multi-task Parameter Efficient Fine-Tuning with a Mixture of Dyadic Experts (2025.findings-naacl)
Copied to clipboard
| Challenge: | Recent efforts have explored mixtures of LoRA modules for multi-task settings, but this study reveals redundancy in the down-projection matrix of these architectures. |
| Approach: | They propose a method to share down-projection matrix across tasks and employ atomic rank-one adapters coupled with routers that allow more sophisticated task-level specialization. |
| Outcome: | The proposed method outperforms state-of-the-art models on a SNI benchmark and provides a practical solution for deploying lightweight models. |
On the Impact of Fine-Tuning on Chain-of-Thought Reasoning (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models have emerged as powerful tools for general intelligence, showcasing advanced natural language processing capabilities. |
| Approach: | They propose to use supervised fine-tuning and Quantized Low-Rank Adapters to improve LLMs' task-specific performance to address privacy and safety risks. |
| Outcome: | The proposed model improves the accuracy of the chain-of-thought reasonings across four datasets and demonstrates that the faithfulness of CoT reasoning decreases. |
RA-LoRA: Rank-Adaptive Parameter-Efficient Fine-Tuning for Accurate 2-bit Quantized Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) with their extensive parameters and high memory demands are challenging to fine-tune for specific applications with limited resources. |
| Approach: | They propose a method that dynamically adjusts the adapter’s rank using rank-subspace analysis, optimizing performance with fewer parameters. |
| Outcome: | The proposed method improves model accuracy with minimal parameter changes and demonstrates the importance of rank dynamics in optimizing quantized LLMs. |
From Bottom to Top: Extending the Potential of Parameter Efficient Fine-Tuning (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to fine-tune large language models primarily focus on the interaction between different layers, ignoring the fact that different layers store different information. |
| Approach: | They propose a Parameter Efficient Fine-Tuning method which freeze pre-trained parameters and fine-tunes only a few task-specific parameters. |
| Outcome: | The proposed methods reduce parameter count to nearly half by omitting fine-tuning in the middle layers. |
Beyond Full Fine-tuning: Harnessing the Power of LoRA for Multi-Task Instruction Tuning (2024.lrec-main)
Copied to clipboard
Chunlei Xin, Yaojie Lu, Hongyu Lin, Shuheng Zhou, Huijia Zhu, Weiqiang Wang, Zhongyi Liu, Xianpei Han, Le Sun
| Challenge: | Low-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning algorithm for large-scale language models. |
| Approach: | They conduct a systematic study of Low-Rank Adaptation (LoRA) on diverse tasks and rich resources with different learning capacities. |
| Outcome: | The proposed algorithm can achieve remarkable performance in high-resource and multi-task scenarios, even comparable to full fine-tuning. |
LoRACoE: Improving Large Language Model via Composition-based LoRA Expert (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show that the Mixture of Experts architecture improves performance of large language models. |
| Approach: | They propose a method to build static experts using LoRA parameters . they propose to use rank-level parameters to build experts based on rank-based parameters based in LoRA module. |
| Outcome: | The proposed method improves task performance across a broader range of tasks. |
Sensitivity-LoRA : Low-Load Sensitivity-Based Fine-Tuning for Large Language Models (2025.findings-emnlp)
Copied to clipboard
Hao Zhang, Bo Huang, Zhenjia Li, Xi Xiao, Hui Yi Leong, Zumeng Zhang, Xinwei Long, Tianyang Wang, Hao Xu
| Challenge: | Low-Rank Adaptation (LoRA) is a promising approach to adapting LLMs to specialized tasks . existing rank allocation techniques remain computationally inefficient and unstable . |
| Approach: | They propose a low-rank adapted model that approximates model weight updates using low-ranked decomposition. |
| Outcome: | The proposed method is limited by its uniform rank allocation to each incremental matrix . it leverages the second-order derivatives of the loss function to capture weight sensitivity . |
G-LoRA: Global-Local Decoupled Low-Rank Adaptation (2026.findings-acl)
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) improves the fine-tuning efficiency and performance of large language models. |
| Approach: | They propose a low-rank adaptive approach that decomposes update matrix into global and local adapters and assigns them to local and global adapters. |
| Outcome: | The proposed method achieves up to 2.7% accuracy improvement over LoRA and its variants on commonsense reasoning, mathematical reasoning, and code generation. |
MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Low-rank adaptation and its mixture-of-experts (MOE) methods are highly effective but introduce significant latency in multi-tenant settings due to the LoRA modules and MOE routers added to multiple linear modules. |
| Approach: | They propose a low-rank adaptation variant that considers each LoRA module as an expert and employs a prompt-aware routing mechanism. |
| Outcome: | Extensive analysis on commonsense reasoning tasks and math reasoning tasks show that MiLoRA outperforms strong PEFT baselines with comparable tunable parameter budgets. |