Abhilasha Lodha, Gayatri Belapurkar, Saloni Chalkapurkar, Yuanming Tao, Reshmi Ghosh, Samyadeep Basu, Dmitrii Petrov, Soundararajan Srinivasan
| Challenge: | preserving knowledge of target distribution by fine-tuning all layers can be expensive and may increase data volume requirements. |
| Approach: | They propose an efficient metric based on the diagonal of the Fisher information matrix (FIM score) to select the candidate layers for selective fine-tuning. |
| Outcome: | The proposed metric can select layers leading to strong performance on GLUE and SuperGLUE tasks and across distinct language encoders. |
Similar Papers
Parameter-Efficient Fine-Tuning: Is There An Optimal Subset of Parameters to Tune? (2024.findings-eacl)
Copied to clipboard
| Challenge: | Recent research has illuminated the possibility of selective parameter-efficient fine-tuning, which retains the inference speed of the original model and comes at no additional computational cost. |
| Approach: | They propose to selectively update only a small subset of parameters during the fine-tuning process, keeping the remaining parameters frozen during training. |
| Outcome: | The proposed methods retain the inference speed of the original model and come at no additional computational cost. |
Exploring Intrinsic Language-specific Subspaces in Fine-tuning Multilingual Neural Machine Translation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual neural machine translation models support fine-tuning hundreds of languages simultaneously. |
| Approach: | They propose to fine-tune a language in its intrinsic subspace with a tiny fraction of entire parameters. |
| Outcome: | The proposed methods outperform full-parameter fine-tuning up to 2.25 spBLEU scores and reduce trainable parameters to 0.4% for high and medium-resource languages and 1.6% for low-resourced ones. |
Layer by Layer: Uncovering Where Multi-Task Learning Happens in Instruction-Tuned Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained large language models retain task-specific knowledge, but where and to what extent they retain it remains unexplored. |
| Approach: | They investigate the task-specific information encoded in pre-trained LLMs and the effects of instruction tuning on their representations across over 60 NLP tasks. |
| Outcome: | The results show that pre-trained models retain task-specific knowledge . some tasks are already encoded in pre-train models, but others benefit from instruction tuning. |
Towards Adaptive Prefix Tuning for Parameter-Efficient Language Model Fine-tuning (2023.acl-short)
Copied to clipboard
| Challenge: | Parameter-efficient fine-tuning only optimizes a few task-specific parameters with frozen pre-trained model. |
| Approach: | They propose to optimize a prefix vector inserted into Transformer layers to optimize the prefix . they propose to use a gate mechanism to adjust the prefixed to each layer . |
| Outcome: | The proposed approach improves on the SuperGLUE and NER datasets. |
G-Tuning: Improving Generalization of Pre-trained Language Models with Generative Adversarial Network (2023.findings-acl)
Copied to clipboard
| Challenge: | Empirical evaluations on the GLUE benchmark demonstrate that fine-tuning can enhance the generalization performance of pre-trained language models (PLMs) in downstream tasks. |
| Approach: | They propose a fine-tuning framework that transforms the latent representation of pre-trained language models from a universal space to a target space and integrates a generative adversarial network into the fine-untun process. |
| Outcome: | Empirical evaluations on the GLUE benchmark and two additional demanding scenarios show that the proposed framework can improve the generalization performance of pre-trained language models (PLMs) in downstream tasks. |
Efficient Layer-wise LLM Fine-tuning for Revision Intention Prediction (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models have shown extraordinary success across text generation tasks . however, their potential for simple yet essential text classification remains underexplored . |
| Approach: | a plug-and-play layer-wise parameter-efficient fine-tuning framework is proposed . it fine- tunes a subset of important LLM layers while freezing redundant ones . |
| Outcome: | a plug-and-play framework fine-tunes a subset of important LLM layers while freezing redundant layers. |
Modular and Parameter-Efficient Fine-Tuning for NLP Models (2022.emnlp-tutorials)
Copied to clipboard
| Challenge: | State-of-the-art language models in NLP perform best when fine-tuned even on small datasets. |
| Approach: | They provide an overview of parameter-efficient fine-tuning methods and highlight similarities and differences . they highlight benefits and usage scenarios of a neglected property of parameter efficient models . |
| Outcome: | This paper provides an overview of parameter-efficient fine-tuning methods . it highlights similarities and differences by presenting them in a unified view . |
Fine-tuning Happens in Tiny Subspaces: Exploring Intrinsic Task-specific Subspaces of Pre-trained Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models are overly parameterized and have significant redundancy . recent studies show that PLMs are highly over-parameterized and robust to pruning . |
| Approach: | They propose to re-parameter and fine-tune pre-trained language models from a new perspective: Discovery of intrinsic task-specific subspace. |
| Outcome: | The proposed model can be fine-tuned in the subspace with a small number of free parameters. |
Scalable Fine-tuning from Multiple Data Sources: A First-Order Approximation Approach (2024.findings-emnlp)
Copied to clipboard
| Challenge: | a new algorithm to estimate fine-tuning performance for a target task is proposed . conventional subset selection methods require repeated training on subsets of auxiliary tasks . |
| Approach: | They propose an algorithm to fine-tune a language model for a target task by optimally using auxiliary tasks' information. |
| Outcome: | The proposed method can estimate fine-tuning performance on CPUs in seconds. |
LexFit: Lexical Fine-Tuning of Pretrained Language Models (2021.acl-long)
Copied to clipboard
| Challenge: | Transformer-based language models implicitly store a wealth of lexical semantic knowledge, but it is non-trivial to extract that knowledge effectively from their parameters. |
| Approach: | They propose to expose and enrich lexical knowledge from transformer-based language models to serve as effective decontextualized word encoders even when fed input words "in isolation" |
| Outcome: | The proposed model outperforms standard static WEs and vanilla LMs in lexical tasks over four established tasks in 8 languages. |