Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks (2021.acl-long)
Copied to clipboard
| Challenge: | State-of-the-art parameter-efficient fine-tuning methods rely on introducing adapter modules between the layers of a pretrained language model. |
| Approach: | They propose a framework that can learn adapter parameters for all layers and tasks by generating them using shared hypernetworks. |
| Outcome: | The proposed framework improves performance on the well-known GLUE benchmark while adding only 0.29% parameters per task. |
Similar Papers
Jointly Reparametrized Multi-Layer Adaptation for Efficient and Private Tuning (2023.findings-acl)
Copied to clipboard
| Challenge: | Efficient finetuning of pretrained language transformers requires a large number of tunable parameters. |
| Approach: | They propose a language transformer finetuning strategy that introduces task-specific parameters in multiple transformer layers. |
| Outcome: | The proposed method outperforms other methods with 4,100 parameters on GLUE tasks with 5% of full finetuning performance. |
AdapterHub: A Framework for Adapting Transformers (2020.emnlp-demos)
Copied to clipboard
Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, Iryna Gurevych
| Challenge: | AdapterHub framework enables dynamic “stiching-in” of pre-trained adapters for different tasks and languages. |
| Approach: | They propose a framework that allows dynamic "stiching-in" of pre-trained adapters for different tasks and languages. |
| Outcome: | The proposed framework allows dynamic “stiching-in” of pre-trained adapters for different tasks and languages. |
Hit the Nail on the Head: Parameter-Efficient Multi-task Tuning via Human Language Intervention (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that PEFT on small pre-trained language models improves multitasking capabilities. |
| Approach: | They propose a multi-task learning framework that enables transfer of prior knowledge across tasks . they attach task descriptions to input samples and map them to task embeddings . |
| Outcome: | The proposed method improves performance on a T5 model and in decoder-only models . |
HyperPELT: Unified Parameter-Efficient Language Model Tuning for Both Language and Vision-and-Language Tasks (2023.findings-acl)
Copied to clipboard
| Challenge: | Pretraining and fine-tuning are the dominant paradigms in natural language processing. |
| Approach: | They propose a parameter-efficient multitask learning framework that takes trainable hyper-embeddings and visual modality as input and outputs weights for different modules in a pretrained language model. |
| Outcome: | The proposed framework adds fewer trainable parameters in multi-task learning while achieving superior performances and transfer ability compared to state-of-the-art methods. |
An Empirical Study on the Transferability of Transformer Modules in Parameter-efficient Fine-tuning (2022.emnlp-main)
Copied to clipboard
| Challenge: | Parameter-efficient fine-tuning is a computationally expensive process . introducing new parameters to an already-large model can be considered a drawback. |
| Approach: | They investigate the capability of different transformer modules in transferring knowledge from a pre-trained model to a downstream task. |
| Outcome: | The proposed methods show that each transformer module is a winning ticket . they show that with only 0.003% updateable parameters, they can show acceptable performance on target tasks. |
LayerConnect: Hypernetwork-Assisted Inter-Layer Connector to Enhance Parameter Efficiency (2022.coling-1)
Copied to clipboard
| Challenge: | Existing parameter-efficient methods focus on reducing trainable parameters but neglect the inference speed, which limits the ability to deploy PLMs. |
| Approach: | They propose to use a hypernetwork-assisted inter-layer connector to enhance inference efficiency by tuning parameters inside a linear connector between two Transformer layers. |
| Outcome: | The proposed model reduces model parameters to 11.75% while preserving performance degradation to less than 5%. |
DimA: A Parameter-efficient Fine-tuning Method with Knowledge Transfer Based on Transformer (2024.lrec-main)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) have demonstrated impressive performance across various downstream tasks, but fine-tuning is computationally expensive and storage-intensive. |
| Approach: | They propose a parameter-efficient method called DimA which enhances the transformer architecture by increasing the dimensionality. |
| Outcome: | The proposed method achieves state-of-the-art results in GLUE and XSUM tasks while utilizing less than 1% of the original model’s parameters. |
Hyperdecoders: Instance-specific decoders for multi-task NLP (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work in NLP has examined the performance of large pretrained transformer-based models in multi-task settings, where a single model is evaluated on multiple tasks simultaneously. |
| Approach: | They propose a method for multi-tasking using a hypernetwork conditioned on the output of an encoder to generate a unique decoder adaptation for every input instance. |
| Outcome: | The proposed method outperforms previous methods for efficient multi-task fine-tuning and maps from encoder representations to output labels. |
Parameter-Efficient Tuning with Special Token Adaptation (2023.eacl-main)
Copied to clipboard
| Challenge: | a recent study shows that parameter-efficient tuning is a challenge for multitask deployments. |
| Approach: | They propose a parameter-efficient tuning technique that only updates a small subset of parameters when adapting a pretrained model to downstream tasks. |
| Outcome: | The proposed method achieves comparable performance to fine-tuning in natural language understanding tasks including text classification and NER with only 0.029% of parameters trained. |
Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformers (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent improvements in NLP tasks can be attributed to the Transformer model. |
| Approach: | They propose to use parameter-sharing methods to reduce parameter budgets in generative models by using sandwich-style parameter sharing and self-attentive embedding factorization. |
| Outcome: | The proposed model outperforms the current RNN model even with significantly fewer parameters. |