NLoPT: N-gram Enhanced Low-Rank Task Adaptive Pre-training for Efficient Language Model Adaption (2024.lrec-main)
Copied to clipboard
| Challenge: | Pre-trained Language Models (PLMs) have superior performance on downstream tasks . however, conventional TAPT adjusts all parameters of the PLMs, which distorts the learned generic knowledge embedded in the original PLM's weights. |
| Approach: | They propose a two-step n-gram enhanced low-rank task adaptive pre-training method to customize a PLM to the downstream task. |
| Outcome: | The proposed method improves performance on six datasets from four domains. |
Similar Papers
Taming Pre-trained Language Models with N-gram Representations for Low-Resource Domain Adaptation (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods to train pre-trained models require domain-specific data and computational resources. |
| Approach: | They propose a domain-aware N-gram Adaptor to incorporate unseen and domain-specific words into a generic pretrained model. |
| Outcome: | The proposed model can improve on eight low-resource tasks using limited data with lower computational costs. |
CoLA: Collaborative Low-Rank Adaptation (2025.findings-acl)
Copied to clipboard
| Challenge: | The scaling law of Large Language Models (LLMs) reveals diminishing return on performance as model scale increases. |
| Approach: | They propose a more flexible LoRA architecture with an efficient initialization scheme . they propose combining three collaborative strategies to enhance performance . |
| Outcome: | The proposed model outperforms existing methods in low-sample scenarios. |
TLoRA: Task-aware Low Rank Adaptation of Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing low-rank Adaptation (LoRA) methods address only one factor, often at the cost of increased training complexity or reduced practical efficiency. |
| Approach: | They propose a low-rank Adaptation framework that optimizes initialization and resource allocation at the outset of training. |
| Outcome: | The proposed framework performs excellently across various tasks while reducing the number of trainable parameters. |
Attention Fusion: a light yet efficient late fusion mechanism for task adaptation in NLU (2022.findings-naacl)
Copied to clipboard
| Challenge: | a recent study has shown that fine-tuning pre-trained models is parameter-inefficient and expensive. |
| Approach: | They propose a task-attuned token module which integrates pre-trained network representations into a pre-trainer. |
| Outcome: | The proposed model trains only 0.0009% of the parameters and is efficient during computation and scalable during deployment. |
PRILoRA: Pruned and Rank-Increasing Low-Rank Adaptation (2024.findings-eacl)
Copied to clipboard
| Challenge: | Several approaches to parameter-efficient fine-tuning have been proposed . low-rank Adaptation (LoRA) does not consider the varying importance of each layer . |
| Approach: | They propose a method that allocates a different rank for each layer and performs pruning throughout the training process. |
| Outcome: | The proposed method is based on eight GLUE benchmarks and is currently the state of the art. |
Systematic Analysis for Pretrained Language Model Priming for Parameter-Efficient Fine-tuning (2024.naacl-srw)
Copied to clipboard
| Challenge: | Parameter-efficient (PE) methods for adapting pre-trained language models to downstream tasks are still lacking in many cases. |
| Approach: | They propose a general PE priming framework to enhance few-shot adaptation and generalization ability of PE methods. |
| Outcome: | The proposed framework reveals that the best priming strategy facilitates adaptation to target tasks. |
Towards Simple and Efficient Task-Adaptive Pre-training for Text Classification (2022.aacl-short)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models are extensively trained on massive heterogeneous datasets, known as pre-training datasets. |
| Approach: | They propose to use Domain Adaptive Pre-training and Task-Adaptive pre-training as intermediate steps before the final finetuning task to cover the target domain vocabulary. |
| Outcome: | The proposed approach is computationally efficient, with 78% fewer parameters trained during TAPT. |
Integrating Task Specific Information into Pretrained Language Models for Low Resource Fine Tuning (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing pretrained language models are agnostic to downstream information and can overfit when fine-tuned with low resource datasets. |
| Approach: | They integrate label information as a task-specific prior into the self-attention component of pretrained BERT models. |
| Outcome: | Experiments on benchmarks and real-word datasets show that the proposed approach can improve the performance of pretrained models when fine-tuned with small datasets. |
Multi-Stage Pre-training for Low-Resource Domain Adaptation (2020.emnlp-main)
Copied to clipboard
Rong Zhang, Revanth Gangi Reddy, Md Arafat Sultan, Vittorio Castelli, Anthony Ferritto, Radu Florian, Efsun Sarioglu Kayi, Salim Roukos, Avi Sil, Todd Ward
| Challenge: | Existing approaches to transfer learning target data to in-domain text . prior work has adapted pre-trained LMs to specific domains . |
| Approach: | They extend the vocabulary of a pretrained language model with domain-specific terms to create synthetic tasks that help it transfer to downstream tasks. |
| Outcome: | The proposed approaches show significant performance gains on extractive reading comprehension, document ranking and duplicate question detection tasks. |
G-LoRA: Global-Local Decoupled Low-Rank Adaptation (2026.findings-acl)
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) improves the fine-tuning efficiency and performance of large language models. |
| Approach: | They propose a low-rank adaptive approach that decomposes update matrix into global and local adapters and assigns them to local and global adapters. |
| Outcome: | The proposed method achieves up to 2.7% accuracy improvement over LoRA and its variants on commonsense reasoning, mathematical reasoning, and code generation. |