Parameter-efficient Tuning for Large Language Model without Calculating Its Gradients (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent parameter-efficient tuning methods can only save 30% of training memory . gradient computation and backpropagation are still necessary for these methods . |
| Approach: | They propose a parameter-efficient tuning method that can be used to fine-tune large language models without calculating gradients. |
| Outcome: | The proposed method saves 30% of training memory and improves performance on large language models. |
Similar Papers
Full Parameter Fine-tuning for Large Language Models with Limited Resources (2024.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) require massive GPU resources for training. |
| Approach: | They propose a parameter-efficient optimization that fuses the gradient computation and parameter update in one step to reduce memory usage. |
| Outcome: | The proposed method reduces memory usage to 10.8% compared to the standard approach. |
LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models (2023.emnlp-main)
Copied to clipboard
Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, Roy Lee
| Challenge: | Large language models (LLMs) have shown unprecedented performance across various tasks. |
| Approach: | They propose an easy-to-use framework that integrates adapters into LLMs . they evaluate adapters on 14 datasets from two different reasoning tasks . |
| Outcome: | The proposed framework can be used to fine-tune open-access language models with task-specific data and instruction data. |
Stochastic Fine-Tuning of Language Models Using Masked Gradients (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are the dominant paradigm in Natural Language Processing but fine-tuning them for specific downstream tasks often requires updating a vast number of parameters. |
| Approach: | They propose a method that selectively updates a small subset of parameters in each step of the tuning process. |
| Outcome: | The proposed approach outperforms existing fine-tuning methods while updating merely **0.08**% of the model’s parameters. |
A Study of Parameter Efficient Fine-tuning by Learning to Efficiently Fine-Tune (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for parameter-efficient fine-tuning (PEFT) are limited due to the need for increased computational resources. |
| Approach: | They propose a method to learn PEFT parameters from data by projecting high dimensional parameters onto low dimensional parameter manifolds or identifying PEFT parametrically. |
| Outcome: | The proposed method can be used to identify PEFT parameters on GLUE tasks. |
Completely Modular Fine-tuning for Dynamic Language Adaptation (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing studies on multilingual fine-tuning with a fixed set of languages lack dynamic adaptability to new languages. |
| Approach: | They propose a modular fine-tuning pipeline that enables dynamic language adaptation for LLMs by first training English-centric adapters for each language separately and then merging them for arbitrary-direction translation. |
| Outcome: | The proposed pipeline achieves 86% performance over traditional fine-tuning on four languages, while training only 0.1% parameters and relying on English as a bridge language without catastrophic forgetting. |
Fine-tuning Large Language Models with Limited Data: A Survey and Practical Guide (2026.tacl-1)
Copied to clipboard
| Challenge: | Pre-trained language models provide strong foundations, but effective adaptation under data scarcity requires efficient and efficient fine-tuning techniques. |
| Approach: | They propose to review parameter-efficient fine-tuning techniques that lower training and deployment costs and domain and cross-lingual adaptation methods for both encoder and decoder models. |
| Outcome: | The proposed techniques lower training and deployment costs, domain and cross-lingual adaptation methods, and model specialization strategies. |
Let’s Focus on Neuron: Neuron-Level Supervised Fine-tuning for Large Language Model (2025.coling-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are composed of neurons that exhibit diverse behaviors and roles. |
| Approach: | They propose a novel approach that refines the granularity of parameter training down to the individual neuron, enabling a more parameter-efficient fine-tuning model. |
| Outcome: | The proposed approach exceeds the performance of full-parameter fine-tuning and PEFT and provides insights into the analysis of neurons. |
Towards Adaptive Prefix Tuning for Parameter-Efficient Language Model Fine-tuning (2023.acl-short)
Copied to clipboard
| Challenge: | Parameter-efficient fine-tuning only optimizes a few task-specific parameters with frozen pre-trained model. |
| Approach: | They propose to optimize a prefix vector inserted into Transformer layers to optimize the prefix . they propose to use a gate mechanism to adjust the prefixed to each layer . |
| Outcome: | The proposed approach improves on the SuperGLUE and NER datasets. |
Modular and Parameter-Efficient Fine-Tuning for NLP Models (2022.emnlp-tutorials)
Copied to clipboard
| Challenge: | State-of-the-art language models in NLP perform best when fine-tuned even on small datasets. |
| Approach: | They provide an overview of parameter-efficient fine-tuning methods and highlight similarities and differences . they highlight benefits and usage scenarios of a neglected property of parameter efficient models . |
| Outcome: | This paper provides an overview of parameter-efficient fine-tuning methods . it highlights similarities and differences by presenting them in a unified view . |
FanLoRA: Fantastic LoRAs and Where to Find Them in Large Language Model Fine-tuning (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Lowrank adaptation and its variants introduce significant latency in multi-tenant settings, hindering their applications in the industry. |
| Approach: | They propose a framework to fine-tune LoRA modules on a large-scale instruction tuning dataset. |
| Outcome: | The proposed framework outperforms existing PEFT methods and significantly reduces inference latency. |