Jointly Reparametrized Multi-Layer Adaptation for Efficient and Private Tuning (2023.findings-acl)
Copied to clipboard
| Challenge: | Efficient finetuning of pretrained language transformers requires a large number of tunable parameters. |
| Approach: | They propose a language transformer finetuning strategy that introduces task-specific parameters in multiple transformer layers. |
| Outcome: | The proposed method outperforms other methods with 4,100 parameters on GLUE tasks with 5% of full finetuning performance. |
Similar Papers
Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks (2021.acl-long)
Copied to clipboard
| Challenge: | State-of-the-art parameter-efficient fine-tuning methods rely on introducing adapter modules between the layers of a pretrained language model. |
| Approach: | They propose a framework that can learn adapter parameters for all layers and tasks by generating them using shared hypernetworks. |
| Outcome: | The proposed framework improves performance on the well-known GLUE benchmark while adding only 0.29% parameters per task. |
BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models (2022.acl-short)
Copied to clipboard
| Challenge: | Large pre-trained models are expensive to train and deploy . large data sets make finetuning expensive to deploy - a new paradigm . |
| Approach: | They propose a sparse-finetuning method where only bias-terms are modified . they show that applying BitFit on pre-trained BERT models is competitive . |
| Outcome: | The proposed method is competitive with sparse-finetuning methods on small-to-medium training data. |
Multi-Task Pre-Finetuning of Lightweight Transformer Encoders for Text Classification and NER (2025.emnlp-industry)
Copied to clipboard
| Challenge: | nave multitask pre-finetuning introduces conflicting optimization signals that degrade overall performance. |
| Approach: | They propose a framework that enables a single shared encoder backbone with modular adapters. |
| Outcome: | The proposed framework achieves comparable performance to individual pre-finetuning while meeting practical deployment constraint. |
MAPLE: Multilingual Evaluation of Parameter Efficient Finetuning of Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Prior work on multilingual evaluation has shown that there is a large gap between the performance of Large Language Models on English and other languages. |
| Approach: | They propose to finetune Llama-2 and Mistral models on two datasets to determine their effect on model performance on six downstream tasks covering forty one languages. |
| Outcome: | The proposed model can improve on six multilingual tasks while degrading on high-resource languages. |
Parameter-Efficient Tuning with Special Token Adaptation (2023.eacl-main)
Copied to clipboard
| Challenge: | a recent study shows that parameter-efficient tuning is a challenge for multitask deployments. |
| Approach: | They propose a parameter-efficient tuning technique that only updates a small subset of parameters when adapting a pretrained model to downstream tasks. |
| Outcome: | The proposed method achieves comparable performance to fine-tuning in natural language understanding tasks including text classification and NER with only 0.029% of parameters trained. |
AlphaTuning: Quantization-Aware Parameter-Efficient Adaptation of Large-Scale Pre-Trained Language Models (2022.findings-emnlp)
Copied to clipboard
Se Jung Kwon, Jeonghoon Kim, Jeongin Bae, Kang Min Yoo, Jin-Hwa Kim, Baeseong Park, Byeongwook Kim, Jung-Woo Ha, Nako Sung, Dongsoo Lee
| Challenge: | Existing approaches to improve inference efficiency by accelerating model fine-tuning have not been thoroughly explored. |
| Approach: | They propose to combine parameter-efficient adaptation and model compression to accelerate model . they propose to freeze binary parameters and scale scaling factors for target tasks . |
| Outcome: | The proposed algorithm achieves >10x compression ratio under 4-bit quantization and >1,000x reduction in trainable parameters. |
Efficient Learning of Multiple NLP Tasks via Collective Weight Factorization on BERT (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods to fine-tune a model for multiple tasks require a large amount of memory and computing power. |
| Approach: | They propose to factorize the weighs of a pre-trained Transformer model to improve training efficiency across multiple tasks by using BERT-Large as an instantiation of the Transformer and the GLUE as the evaluation benchmark. |
| Outcome: | The proposed method matches or improves the original fine-tuned model’s performance for each task while effectively decreasing parameter requirements by two orders of magnitude. |
Towards Adaptive Prefix Tuning for Parameter-Efficient Language Model Fine-tuning (2023.acl-short)
Copied to clipboard
| Challenge: | Parameter-efficient fine-tuning only optimizes a few task-specific parameters with frozen pre-trained model. |
| Approach: | They propose to optimize a prefix vector inserted into Transformer layers to optimize the prefix . they propose to use a gate mechanism to adjust the prefixed to each layer . |
| Outcome: | The proposed approach improves on the SuperGLUE and NER datasets. |
A Semantic-Aware Layer-Freezing Approach to Computation-Efficient Fine-Tuning of Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing work on how to finetune but neglects the issue of where to fine-tune language models is expensive. |
| Approach: | They propose to use transition traces of latent representation to compute deviations (or loss) and then estimate the gain of each layer in reducing deviation (or gain). |
| Outcome: | The proposed approach outperforms baseline methods and is cost-benefit balanced. |
Exploring Intrinsic Language-specific Subspaces in Fine-tuning Multilingual Neural Machine Translation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual neural machine translation models support fine-tuning hundreds of languages simultaneously. |
| Approach: | They propose to fine-tune a language in its intrinsic subspace with a tiny fraction of entire parameters. |
| Outcome: | The proposed methods outperform full-parameter fine-tuning up to 2.25 spBLEU scores and reduce trainable parameters to 0.4% for high and medium-resource languages and 1.6% for low-resourced ones. |