FaLA: Fast Linear Adaptation for Replacing Backbone Models on Edge Devices (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Current NLP models heavily rely on pre-trained models, such as BERT and RoBERTa. |
| Approach: | They propose a lightweight method for personalized NLP classification tasks post-backbone replacement using a personalized matrix calculated from documents corresponding to users' old and new backbones. |
| Outcome: | The proposed method achieves over 1000 times computation reduction in Flops for backpropagation and brings the user-specific initialization for personal matrix yielding significant performance boost compared with popular transfer learning methods. |
Similar Papers
How to Make LMs Strong Node Classifiers? (2026.findings-eacl)
Copied to clipboard
Zhe Xu, Kaveh Hassani, Si Zhang, Hanqing Zeng, Michihiro Yasunaga, Limei Wang, Dongqi Fu, Ning Yao, Bo Long, Hanghang Tong
| Challenge: | Language Models (LMs) are increasingly challenging the dominance of domain-specific models, such as Graph Neural Networks (GNNs) and Graph Transformers (GTs). |
| Approach: | They propose a novel approach that empowers off-the-shelf LMs to achieve performance comparable to state-of-the art (SOTA) GNNs on node classification tasks without requiring any architectural modifications. |
| Outcome: | The proposed approach outperforms existing GNNs on node classification tasks and is open-source upon publication. |
SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning pre-trained models fail to generalize to unseen data. |
| Approach: | They propose a framework for robust and efficient fine-tuning for pre-trained models . proposed framework achieves new state-of-the-art performance on a number of NLP tasks . |
| Outcome: | The proposed framework outperforms the state-of-the-art T5 model on GLUE, SNLI, SciTail and ANLI. |
Instant Personalized Large Language Model Adaptation via Hypernetwork (2026.acl-long)
Copied to clipboard
Zhaoxuan Tan, Zixuan Zhang, Haoyang Wen, Zheng Li, Rongzhi Zhang, Pei Chen, Fengran Mo, Zheyuan Liu, Qingkai Zeng, Qingyu Yin, Meng Jiang
| Challenge: | Existing parameter-efficient fine-tuning methods require training a separate adapter for each user, making them computationally expensive and impractical for real-time updates. |
| Approach: | They propose a scalable framework that maps a user's profile directly to a full set of adapter parameters. |
| Outcome: | The proposed framework outperforms prompt-based personalization and OPPU while using substantially fewer computational resources at deployment. |
Parameter-Efficient Tuning Makes a Good Classification Head (2022.emnlp-main)
Copied to clipboard
| Challenge: | In recent years, pretrained models revolutionized the paradigm of natural language understanding . but the final-layer output of the backbone, i.e. the input of the classification head, will change greatly during finetuning . |
| Approach: | They propose to append a randomly initialized classification head after the pretrained backbone and finetune the whole model. |
| Outcome: | The proposed classification head can be replaced with the randomly initialized heads for a stable performance gain. |
Robust Transfer Learning with Pretrained Language Models through Adapters (2021.acl-short)
Copied to clipboard
| Challenge: | Existing approaches to transfer learning with pretrained transformer-based language models are not robust and can be adversarial. |
| Approach: | They propose a simple yet effective adapter-based approach to fine-tune language models on downstream tasks. |
| Outcome: | The proposed approach improves stability and adversarial robustness in transfer learning to various downstream tasks. |
Training Text-to-Text Transformers with Privacy Guarantees (2022.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in NLP often stem from large transformer-based pre-trained models. |
| Approach: | They propose differentially private (DP) training as a potential mitigation for models that can memorize parts of training data. |
| Outcome: | The proposed model can memorize parts of training data and mitigate memorization concerns. |
Know Where You’re Going: Meta-Learning for Parameter-Efficient Fine-Tuning (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on parameter-efficient fine-tuning methods require additional measures after pre-training and before fine-uning. |
| Approach: | They propose to take parameter-efficient fine-tuning into consideration after pre-training and before fine-uning and use meta-learning to prime a model specifically for parameter-efficiency. |
| Outcome: | The proposed method improves on a pre-trained model with certain modifications and achieves 4.96 points on cross-lingual NER fine-tuning. |
Adapt Once, Thrive with Updates: Transferable Parameter-Efficient Fine-Tuning on Evolving Base Models (2025.acl-long)
Copied to clipboard
| Challenge: | Parameter-efficient fine-tuning (PEFT) is a common method for fine- tuning large language models . however, once updated, PEFT modules suffer performance degradation on newer versions . |
| Approach: | They propose a method that enhances the PEFT module by focusing on the task-specific pattern while reducing its dependence on certain knowledge in the base model. |
| Outcome: | Experiments show that PEFT modules can maintain performance on updated models without re-tuning . the proposed approach can be used in real-world applications with large model sizes . |
Parameter-Efficient Tuning with Special Token Adaptation (2023.eacl-main)
Copied to clipboard
| Challenge: | a recent study shows that parameter-efficient tuning is a challenge for multitask deployments. |
| Approach: | They propose a parameter-efficient tuning technique that only updates a small subset of parameters when adapting a pretrained model to downstream tasks. |
| Outcome: | The proposed method achieves comparable performance to fine-tuning in natural language understanding tasks including text classification and NER with only 0.029% of parameters trained. |
PRILoRA: Pruned and Rank-Increasing Low-Rank Adaptation (2024.findings-eacl)
Copied to clipboard
| Challenge: | Several approaches to parameter-efficient fine-tuning have been proposed . low-rank Adaptation (LoRA) does not consider the varying importance of each layer . |
| Approach: | They propose a method that allocates a different rank for each layer and performs pruning throughout the training process. |
| Outcome: | The proposed method is based on eight GLUE benchmarks and is currently the state of the art. |