HyperPELT: Unified Parameter-Efficient Language Model Tuning for Both Language and Vision-and-Language Tasks (2023.findings-acl)
Copied to clipboard
| Challenge: | Pretraining and fine-tuning are the dominant paradigms in natural language processing. |
| Approach: | They propose a parameter-efficient multitask learning framework that takes trainable hyper-embeddings and visual modality as input and outputs weights for different modules in a pretrained language model. |
| Outcome: | The proposed framework adds fewer trainable parameters in multi-task learning while achieving superior performances and transfer ability compared to state-of-the-art methods. |
Similar Papers
Making Parameter-efficient Tuning More Efficient: A Unified Framework for Classification Tasks (2022.coling-1)
Copied to clipboard
Xin Zhou, Ruotian Ma, Yicheng Zou, Xuanting Chen, Tao Gui, Qi Zhang, Xuanjing Huang, Rui Xie, Wei Wu
| Challenge: | Large pre-trained language models (PLMs) have demonstrated superior performance in industrial applications. |
| Approach: | They propose a framework that re-uses existing parameter-efficient methods with a unified classifier. |
| Outcome: | The proposed framework improves the efficiency of existing parameter-efficient methods with a unified classifier. |
Parameter-Efficient Cross-lingual Transfer of Vision and Language Models via Translation-based Alignment (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing cross-lingual transfer methods that use labeled data and linguistic resources would consume excessive resources for a large number of languages. |
| Approach: | They propose a parameter-efficient cross-lingual transfer learning framework that utilizes a translation-based alignment method to mitigate multilingual disparities. |
| Outcome: | The proposed framework reduces disparities among languages and improves cross-lingual transfer results in low-resource scenarios while keeping and fine-tuning only a small number of parameters. |
Hit the Nail on the Head: Parameter-Efficient Multi-task Tuning via Human Language Intervention (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that PEFT on small pre-trained language models improves multitasking capabilities. |
| Approach: | They propose a multi-task learning framework that enables transfer of prior knowledge across tasks . they attach task descriptions to input samples and map them to task embeddings . |
| Outcome: | The proposed method improves performance on a T5 model and in decoder-only models . |
Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Large-scale language models can be fine-tuned to learn highly transferable embedding, but they are expensive and require multiple model parameters. |
| Approach: | They propose a way to fine-tune multiple down-stream generation tasks simultaneously using a single, large pretrained model. |
| Outcome: | The proposed model can maintain or improve the performance of fine-tuning the whole model. |
UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuning (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods for parameter-efficient language model tuning (PELT) match the performance of fine-tuning with fewer trainable parameters. |
| Approach: | They propose a framework which integrates different PELT methods as submodules and learns to activate the ones that best suit the current data or task setup via gating mechanism. |
| Outcome: | The proposed framework outperforms fine-tuning methods on the GLUE benchmark and achieves 14% gains over the best individual PELT method. |
Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks (2021.acl-long)
Copied to clipboard
| Challenge: | State-of-the-art parameter-efficient fine-tuning methods rely on introducing adapter modules between the layers of a pretrained language model. |
| Approach: | They propose a framework that can learn adapter parameters for all layers and tasks by generating them using shared hypernetworks. |
| Outcome: | The proposed framework improves performance on the well-known GLUE benchmark while adding only 0.29% parameters per task. |
MultiFiT: Efficient Multi-lingual Language Model Fine-tuning (D19-1)
Copied to clipboard
| Challenge: | Pretrained language models require unlabelled data for training, while cross-lingual models underperform on low-resource languages. |
| Approach: | They propose a multi-lingual language model fine-tuning to train and fine- tune language models efficiently in their own language. |
| Outcome: | The proposed method outperforms existing models on two widely used datasets on cross-lingual classification tasks. |
UDapter: Language Adaptation for Truly Universal Dependency Parsing (2020.emnlp-main)
Copied to clipboard
| Challenge: | Cross-language interference and restrained model capacity remain major obstacles in multilingual dependency parsing. |
| Approach: | They propose a multilingual task adaptation approach based on contextual parameter generation and adapter modules that learn adapters via language embeddings while sharing model parameters across languages. |
| Outcome: | The proposed approach outperforms strong monolingual and multilingual baselines on most languages on high-resource and low-resourced (zero-shot) languages. |
Hyper-X: A Unified Hypernetwork for Multi-Task Multilingual Transfer (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing multilingual models cannot fully leverage training data when it is available in different task-language combinations. |
| Approach: | They propose a single hypernetwork that unifies multi-task and multilingual learning with efficient adaptation. |
| Outcome: | The proposed model achieves the best or competitive gain when a mixture of multiple resources is available while being significantly more efficient than existing models. |
Modular and Parameter-Efficient Fine-Tuning for NLP Models (2022.emnlp-tutorials)
Copied to clipboard
| Challenge: | State-of-the-art language models in NLP perform best when fine-tuned even on small datasets. |
| Approach: | They provide an overview of parameter-efficient fine-tuning methods and highlight similarities and differences . they highlight benefits and usage scenarios of a neglected property of parameter efficient models . |
| Outcome: | This paper provides an overview of parameter-efficient fine-tuning methods . it highlights similarities and differences by presenting them in a unified view . |