Efficient Ensemble for Fine-tuning Language Models on Multiple Datasets (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning language models are efficient when adapting to a single dataset. |
| Approach: | They propose to use an ensemble method for fine-tuning a language model to multiple datasets instead of a single adapter per task. |
| Outcome: | The proposed method improves performance on multiple datasets while preserving low-rank adaptation properties. |
Similar Papers
Low-Rank Adaptation for Multilingual Summarization: An Empirical Study (2024.findings-naacl)
Copied to clipboard
| Challenge: | Pre-trained Large Language Models have significantly advanced NLP, but their ever-increasing size poses significant challenges for conventional fine-tuning. |
| Approach: | They investigate the potential of Low-Rank Adaptation (LoRA) in multilingual summarization, a task that is challenging and relatively unexplored. |
| Outcome: | The proposed method outperforms full fine-tuning and cross-lingual transfer strategies in multilingual summarization tasks. |
Beyond Full Fine-tuning: Harnessing the Power of LoRA for Multi-Task Instruction Tuning (2024.lrec-main)
Copied to clipboard
Chunlei Xin, Yaojie Lu, Hongyu Lin, Shuheng Zhou, Huijia Zhu, Weiqiang Wang, Zhongyi Liu, Xianpei Han, Le Sun
| Challenge: | Low-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning algorithm for large-scale language models. |
| Approach: | They conduct a systematic study of Low-Rank Adaptation (LoRA) on diverse tasks and rich resources with different learning capacities. |
| Outcome: | The proposed algorithm can achieve remarkable performance in high-resource and multi-task scenarios, even comparable to full fine-tuning. |
Scalable Fine-tuning from Multiple Data Sources: A First-Order Approximation Approach (2024.findings-emnlp)
Copied to clipboard
| Challenge: | a new algorithm to estimate fine-tuning performance for a target task is proposed . conventional subset selection methods require repeated training on subsets of auxiliary tasks . |
| Approach: | They propose an algorithm to fine-tune a language model for a target task by optimally using auxiliary tasks' information. |
| Outcome: | The proposed method can estimate fine-tuning performance on CPUs in seconds. |
QDyLoRA: Quantized Dynamic Low-Rank Adaptation for Efficient Large Language Model Tuning (2024.emnlp-industry)
Copied to clipboard
Hossein Rajabzadeh, Mojtaba Valipour, Tianshu Zhu, Marzieh Tahaei, Hyock Kwon, Ali Ghodsi, Boxing Chen, Mehdi Rezagholizadeh
| Challenge: | Existing methods to fine tune large language models require huge memory, limiting the choice to acquire Larger models. |
| Approach: | They propose an efficient quantization approach for dynamic low-rank adaptation that can efficiently fine tune large language models on a set of pre-defined LoRA ranks. |
| Outcome: | The proposed method outperforms QLoRA and is competitive to QLouRA and outperformed when employing its optimal rank. |
Advancing Vision-Language Models with Adapter Ensemble Strategies (2024.findings-emnlp)
Copied to clipboard
| Challenge: | CLIP revolutes vision-language pretraining by using contrastive learning on paired web data. |
| Approach: | They propose to combine a "adapter ensemble" with traditional machine learning techniques to augment large-scale pretrained vision-language models. |
| Outcome: | The proposed model outperforms baselines and derives improvement when the number of ensemble parameters increases. |
DenseLoRA: Dense Low-Rank Adaptation of Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Low-rank adaptation (LoRA) is an efficient approach for adapting large language models (LLMs) but many of the weights in these matrices are redundant, leading to inefficiencies in parameter utilization. |
| Approach: | They propose a low-rank adaptation approach that fine-tunes two low-ranked matrices and adapts them through a dense low-Rank matrix, improving parameter utilization and adaptation efficiency. |
| Outcome: | The proposed approach achieves 83.8% accuracy with only 0.01% of trainable parameters compared to LoRA's 80.8% with 0.70% of trainability parameters on LLaMA3-8B. |
AlphaLoRA: Assigning LoRA Experts Based on Layer Training Quality (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies combine LoRA with Mixture-of-Experts (MoE) to improve performance in Large Language Models. |
| Approach: | They propose a method to combine LoRA and Mixture-of-Experts (MoE) to improve performance in Large Language Models. |
| Outcome: | The proposed method reduces redundancy in LoRA experts within the MoE architecture, and improves training quality across layers. |
FanLoRA: Fantastic LoRAs and Where to Find Them in Large Language Model Fine-tuning (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Lowrank adaptation and its variants introduce significant latency in multi-tenant settings, hindering their applications in the industry. |
| Approach: | They propose a framework to fine-tune LoRA modules on a large-scale instruction tuning dataset. |
| Outcome: | The proposed framework outperforms existing PEFT methods and significantly reduces inference latency. |
Fine-tuning Large Language Models with Limited Data: A Survey and Practical Guide (2026.tacl-1)
Copied to clipboard
| Challenge: | Pre-trained language models provide strong foundations, but effective adaptation under data scarcity requires efficient and efficient fine-tuning techniques. |
| Approach: | They propose to review parameter-efficient fine-tuning techniques that lower training and deployment costs and domain and cross-lingual adaptation methods for both encoder and decoder models. |
| Outcome: | The proposed techniques lower training and deployment costs, domain and cross-lingual adaptation methods, and model specialization strategies. |
RA-LoRA: Rank-Adaptive Parameter-Efficient Fine-Tuning for Accurate 2-bit Quantized Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) with their extensive parameters and high memory demands are challenging to fine-tune for specific applications with limited resources. |
| Approach: | They propose a method that dynamically adjusts the adapter’s rank using rank-subspace analysis, optimizing performance with fewer parameters. |
| Outcome: | The proposed method improves model accuracy with minimal parameter changes and demonstrates the importance of rank dynamics in optimizing quantized LLMs. |