| Challenge: | Parameter-efficientfinetuning (PEFT) has gained popularity as a lightweight approach for model customization. |
| Approach: | They propose a parameter-efficient dropout method that is overfitting-prone and parameter-freezed. |
| Outcome: | The proposed method is superior to existing methods and compares with transformer-specific methods. |
Similar Papers
LoRA-drop: Efficient LoRA Parameter Pruning based on Output Evaluation (2025.coling-main)
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) is currently the most commonly used PEFT method for fine-tuning models with billions of parameters. |
| Approach: | They propose to use low-rank Adaptation to evaluate LoRA parameter features and then retain LoRA for important layers and the other layers share the same LoRA. |
| Outcome: | The proposed method achieves comparable performance to full fine-tuning and LoRA while retaining 50% of the LoRA parameters on average. |
UniDrop: A Simple yet Effective Technique to Improve Transformer without Extra Cost (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing approaches to improve the performance of natural language processing models are over-parameterized and overfitted. |
| Approach: | They propose an approach to integrate dropout techniques into the training of Transformer models. |
| Outcome: | The proposed approach can achieve 1.5 BLEU improvement on IWSLT14 translation tasks and better accuracy for the classification even using strong pre-trained RoBERTa as backbone. |
NormAL LoRA: What is the perfect size? (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are crucial for enabling intelligent experiences across applications. |
| Approach: | They propose a low-rank adaptive localization method that uses rank-norm regularization to determine the optimal rank for each weight matrix. |
| Outcome: | NormAL LoRA reduces adapter parameters by 37% while preserving full fine-tuning performance. |
Drop Dropout on Single Epoch Language Model Pretraining (2025.findings-acl)
Copied to clipboard
| Challenge: | Initial dropout was seen as a breakthrough regularization technique that reduced overfitting, yet single-epoch pretraining tasks common to modern LLMs yield minimal overfit. |
| Approach: | They propose to use dropout during single-epoch pretraining to reduce overfitting in language modeling, morpho-syntax, question answering, and MNLI to improve performance. |
| Outcome: | The results show that dropout is not used in large LLMs and improves performance in language modeling, morpho-syntax, question answering, and MNLI. |
DenseLoRA: Dense Low-Rank Adaptation of Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Low-rank adaptation (LoRA) is an efficient approach for adapting large language models (LLMs) but many of the weights in these matrices are redundant, leading to inefficiencies in parameter utilization. |
| Approach: | They propose a low-rank adaptation approach that fine-tunes two low-ranked matrices and adapts them through a dense low-Rank matrix, improving parameter utilization and adaptation efficiency. |
| Outcome: | The proposed approach achieves 83.8% accuracy with only 0.01% of trainable parameters compared to LoRA's 80.8% with 0.70% of trainability parameters on LLaMA3-8B. |
Layer-wise Regularized Dropout for Neural Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods to regularize dropout are consistency training and dropout is a problem in many pre-trained neural language models. |
| Approach: | They propose a layer-wise regularized dropout technique which regularizes dropout at the output layer using consistency training. |
| Outcome: | The proposed model can be regarded as a "self-distillation" framework, in which each sub-model generated by dropout is the other's "teacher" model and "student" model. |
Sensitivity-LoRA : Low-Load Sensitivity-Based Fine-Tuning for Large Language Models (2025.findings-emnlp)
Copied to clipboard
Hao Zhang, Bo Huang, Zhenjia Li, Xi Xiao, Hui Yi Leong, Zumeng Zhang, Xinwei Long, Tianyang Wang, Hao Xu
| Challenge: | Low-Rank Adaptation (LoRA) is a promising approach to adapting LLMs to specialized tasks . existing rank allocation techniques remain computationally inefficient and unstable . |
| Approach: | They propose a low-rank adapted model that approximates model weight updates using low-ranked decomposition. |
| Outcome: | The proposed method is limited by its uniform rank allocation to each incremental matrix . it leverages the second-order derivatives of the loss function to capture weight sensitivity . |
FanLoRA: Fantastic LoRAs and Where to Find Them in Large Language Model Fine-tuning (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Lowrank adaptation and its variants introduce significant latency in multi-tenant settings, hindering their applications in the industry. |
| Approach: | They propose a framework to fine-tune LoRA modules on a large-scale instruction tuning dataset. |
| Outcome: | The proposed framework outperforms existing PEFT methods and significantly reduces inference latency. |
R-LoRA: Randomized Multi-Head LoRA for Efficient Multi-task Learning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) improves performance in multi-task learning by diversifying the head matrices through Multi-Head Dropout and Multi-head Random Initialization. |
| Approach: | They propose a low-rank adaptive approach to fine-tune large language models by approximating weight updates through low-ranked matrices. |
| Outcome: | The proposed approach improves performance in multi-task learning while reducing memory usage and training time. |
UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuning (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods for parameter-efficient language model tuning (PELT) match the performance of fine-tuning with fewer trainable parameters. |
| Approach: | They propose a framework which integrates different PELT methods as submodules and learns to activate the ones that best suit the current data or task setup via gating mechanism. |
| Outcome: | The proposed framework outperforms fine-tuning methods on the GLUE benchmark and achieves 14% gains over the best individual PELT method. |