PLEX: Adaptive Parameter-Efficient Fine-Tuning for Code LLMs using Lottery-Tickets (2025.naacl-industry)
Copied to clipboard
| Challenge: | PLEX is a lottery-ticket based parameter-efficient fine-tuning method that adapts large language models to well-supported and underrepresented programming languages (PLs) in pretraining. |
| Approach: | They propose a lottery-ticket based parameter-efficient fine-tuning method that adapts large language models to well-supported and underrepresented programming languages (PLs) |
| Outcome: | The proposed method achieves state-of-the-art performance among PEFT methods while maintaining competitive results with reduced computational overhead. |
Similar Papers
PEFT-Bench: A Parameter-Efficient Fine-Tuning Methods Benchmark (2026.eacl-long)
Copied to clipboard
| Challenge: | Parameter-Efficient Fine-Tuning (PEFT) methods reduce the number of trainable parameters while maintaining strong downstream performance. |
| Approach: | They propose a unified benchmark for evaluating diverse PEFT methods on autoregressive LLMs. |
| Outcome: | The proposed methods reduce trainable parameters while maintaining strong downstream performance. |
A Study of Parameter Efficient Fine-tuning by Learning to Efficiently Fine-Tune (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for parameter-efficient fine-tuning (PEFT) are limited due to the need for increased computational resources. |
| Approach: | They propose a method to learn PEFT parameters from data by projecting high dimensional parameters onto low dimensional parameter manifolds or identifying PEFT parametrically. |
| Outcome: | The proposed method can be used to identify PEFT parameters on GLUE tasks. |
PEFT-Factory: Unified Parameter-Efficient Fine-Tuning of Autoregressive Large Language Models (2026.eacl-demo)
Copied to clipboard
| Challenge: | Parameter-Efficient Fine-Tuning (PEFT) methods address the increasing size of Large Language Models (LLMs). |
| Approach: | They propose a framework for efficient fine-tuning Large Language Models (LLMs) they aim to train only a small percentage of the full model's parameters . |
| Outcome: | Xu et al., 2023; Ding e t al, 2024; Lialin e al. 2023) show that using PEFT methods can improve performance. |
BIPEFT: Budget-Guided Iterative Search for Parameter Efficient Fine-Tuning of Large Pretrained Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for parameter-efficient fine-tuning are limited by computational and storage requirements. |
| Approach: | They propose a budget-guided iterative search strategy to disentangle binary module and rank dimension search spaces and early selection strategies based on parameter budgets. |
| Outcome: | The proposed method significantly improves search efficiency on public benchmarks. |
Propulsion: Steering LLM with Tiny Fine-Tuning (2025.coling-main)
Copied to clipboard
| Challenge: | Propulsion is a parameter-efficient fine-tuning method that selectively re-scales specific dimensions of a pre-trained model without modifying the model’s parameters. |
| Approach: | They propose a parameter-efficient fine-tuning method that selectively re-scales specific dimensions of a pre-trained model without modifying the parameters. |
| Outcome: | The proposed method reduces parameter count from 355.3 million to 0.086 million while maintaining competitive performance across benchmarks. |
Unlocking Parameter-Efficient Fine-Tuning for Low-Resource Language Translation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Parameter-efficient fine-tuning (PEFT) methods are important in low-resource language (LRL) Neural Machine Translation (NMT) but their practical effectiveness varies significantly across different languages. |
| Approach: | They evaluated the performance of 8 parameters-efficient fine-tuning methods with 15 architectures using the SacreBLEU score. |
| Outcome: | The Houlsby+Inversion adapter outperforms the baseline architectures in both in-domain and out-domain tests and the Houlson+Inverter achieves the best performance overall. |
Finding the Dominant Winning Ticket in Pre-Trained Language Models (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on pre-trained language models show that they can fine-tune parameters but achieve good downstream performance. |
| Approach: | They find that a dominant winning ticket takes up 0.05% of the parameters and is transferable across different tasks. |
| Outcome: | The proposed model can achieve comparable performance with the full-parameter model, the authors show . the dominant winning ticket takes up 0.05% of the parameters, and the model is transferable across tasks, they show - the authors conclude . |
Efficient Layer-wise LLM Fine-tuning for Revision Intention Prediction (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models have shown extraordinary success across text generation tasks . however, their potential for simple yet essential text classification remains underexplored . |
| Approach: | a plug-and-play layer-wise parameter-efficient fine-tuning framework is proposed . it fine- tunes a subset of important LLM layers while freezing redundant ones . |
| Outcome: | a plug-and-play framework fine-tunes a subset of important LLM layers while freezing redundant layers. |
PAC-tuning: Fine-tuning Pre-trained Language Models with PAC-driven Perturbed Gradient Descent (2023.emnlp-main)
Copied to clipboard
| Challenge: | PAC-tuning is a two-stage fine-tune method for pretrained language models . PAC training minimizes the PACBayes generalization bound to learn proper parameter distribution . |
| Approach: | They propose a two-stage fine-tuning method to minimize the PAC-Bayes generalization bound . they use PAC to inject noise with variance learned in the first stage into the model parameters . |
| Outcome: | The proposed method outperforms baseline methods on 5 GLUE benchmark tasks. |
LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models (2023.emnlp-main)
Copied to clipboard
Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, Roy Lee
| Challenge: | Large language models (LLMs) have shown unprecedented performance across various tasks. |
| Approach: | They propose an easy-to-use framework that integrates adapters into LLMs . they evaluate adapters on 14 datasets from two different reasoning tasks . |
| Outcome: | The proposed framework can be used to fine-tune open-access language models with task-specific data and instruction data. |