Adaptive LoRA Merge with Parameter Pruning for Low-Resource Generation (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for adapting LLMs to low-resource tasks keep LoRA parameters frozen and the low-level problem out of their scope. |
| Approach: | They propose a LoRA merge method that updates and prunes LoRA parameters through fine-tuning with minimal target task data. |
| Outcome: | The proposed method improves performance on a low-resource language generation task and improves on previous methods. |
Similar Papers
LoRA Soups: Merging LoRAs for Practical Skill Composition Tasks (2025.coling-industry)
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) is a popular technique for parameter-efficient fine-tuning of Large Language Models. |
| Approach: | They propose to combine LoRA modules to achieve skill composition . they propose to use concatenation of LoRAs to optimize weights for different LoRA training . |
| Outcome: | The proposed model outperforms existing models and data- merging techniques on math-word problems and domain-specialized corpora. |
Unraveling LoRA Interference: Orthogonal Subspaces for Robust Model Merging (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning large language models fail due to performance degradation . existing methods fail for models fine- tuned with low-rank adaptation . |
| Approach: | They propose to constrain the LoRA subspace prior to fine-tuning to ensure that updates relevant to one task do not adversely shift outputs for others. |
| Outcome: | The proposed method can integrate with most existing merging algorithms, reducing unintended interference among tasks. |
LoRA on the Go: Instance-level Dynamic LoRA Selection and Merging (2026.acl-long)
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) is a parameter-efficient approach for fine-tuning large language models. |
| Approach: | They propose a low-rank Adaptation framework that automatically selects and merges LoRA adapters at the instance level without additional training. |
| Outcome: | The proposed framework outperforms training-based baselines on some tasks upto a margin of 3.6% while remaining competitive on other tasks and maintaining inference throughput. |
Beyond Full Fine-tuning: Harnessing the Power of LoRA for Multi-Task Instruction Tuning (2024.lrec-main)
Copied to clipboard
Chunlei Xin, Yaojie Lu, Hongyu Lin, Shuheng Zhou, Huijia Zhu, Weiqiang Wang, Zhongyi Liu, Xianpei Han, Le Sun
| Challenge: | Low-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning algorithm for large-scale language models. |
| Approach: | They conduct a systematic study of Low-Rank Adaptation (LoRA) on diverse tasks and rich resources with different learning capacities. |
| Outcome: | The proposed algorithm can achieve remarkable performance in high-resource and multi-task scenarios, even comparable to full fine-tuning. |
Evolutionary Negative Module Pruning for Better LoRA Merging (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for integrating multiple low-rank Adaptation experts into a single backbone are limited by negative modules. |
| Approach: | They propose a plug-and-play LoRA pruning method to locate and exclude negative modules prior to merging. |
| Outcome: | The proposed method boosts the performance of existing merging algorithms across languages and vision domains. |
PRILoRA: Pruned and Rank-Increasing Low-Rank Adaptation (2024.findings-eacl)
Copied to clipboard
| Challenge: | Several approaches to parameter-efficient fine-tuning have been proposed . low-rank Adaptation (LoRA) does not consider the varying importance of each layer . |
| Approach: | They propose a method that allocates a different rank for each layer and performs pruning throughout the training process. |
| Outcome: | The proposed method is based on eight GLUE benchmarks and is currently the state of the art. |
LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Tasks (2024.acl-long)
Copied to clipboard
| Challenge: | LoRA-Flow uses lightweight modules to customize large language models for downstream tasks . previous work on LoRA combination relied on task-level weights for each involved LoRA . |
| Approach: | They propose a LoRA-Flow approach that uses dynamic weights to adjust the impact of different LoRAs. |
| Outcome: | The proposed method outperforms baselines with task-level weights on six generative tasks. |
Unlocking the Potential of Model Merging for Low-Resource Languages (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Adapting large language models (LLMs) to new languages requires continual pre-training followed by supervised fine-tuning. |
| Approach: | They propose a model merging solution that integrates LLMs with distinct capabilities into a single model without additional training. |
| Outcome: | The proposed model merging outperforms CT-then-SFT in low-resource languages with scarce data. |
LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wild (2024.findings-acl)
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) is an effective yet efficient solution for fine-tuning large language models. |
| Approach: | They propose a low-rank Adaptation framework that retrieves and composes multiple LoRAs according to input prompts. |
| Outcome: | Experimental results show that LoraRetriever outperforms baselines in terms of performance and versatility. |
ResLoRA: Identity Residual Mapping in Low-Rank Adaption (2024.findings-acl)
Copied to clipboard
Shuhua Shi, Shaohan Huang, Minghui Song, Zhoujun Li, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang
| Challenge: | Low-rank adaptation (LoRA) is one of the most popular parameter-efficient fine-tuning methods. |
| Approach: | They propose a low-rank adaptation method that adds residual paths during training and merges them together during inference to achieve better results. |
| Outcome: | The proposed method achieves 2.5x faster convergence speed and improves performance by 14.3% on NLG, NLU, and text-to-image tasks. |