Adapters: A Unified Library for Parameter-Efficient and Modular Transfer Learning (2023.emnlp-demo)
Copied to clipboard
Clifton Poth, Hannah Sterz, Indraneil Paul, Sukannya Purkayastha, Leon Engländer, Timo Imhof, Ivan Vulić, Sebastian Ruder, Iryna Gurevych, Jonas Pfeiffer
| Challenge: | Adapters is an open-source library that unifies parameter-efficient and modular transfer learning in large language models. |
| Approach: | They propose to integrate 10 different methods into a unified interface for parameter-efficient and modular transfer learning in large language models. |
| Outcome: | The proposed library is able to perform on multiple NLP tasks and is open-source. |
Similar Papers
LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models (2023.emnlp-main)
Copied to clipboard
Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, Roy Lee
| Challenge: | Large language models (LLMs) have shown unprecedented performance across various tasks. |
| Approach: | They propose an easy-to-use framework that integrates adapters into LLMs . they evaluate adapters on 14 datasets from two different reasoning tasks . |
| Outcome: | The proposed framework can be used to fine-tune open-access language models with task-specific data and instruction data. |
AdapterHub: A Framework for Adapting Transformers (2020.emnlp-demos)
Copied to clipboard
Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, Iryna Gurevych
| Challenge: | AdapterHub framework enables dynamic “stiching-in” of pre-trained adapters for different tasks and languages. |
| Approach: | They propose a framework that allows dynamic "stiching-in" of pre-trained adapters for different tasks and languages. |
| Outcome: | The proposed framework allows dynamic “stiching-in” of pre-trained adapters for different tasks and languages. |
Adaptable Adapters (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing work uses the same adapter architecture for every dataset regardless of the properties of the dataset or the amount of training data. |
| Approach: | They propose to use adaptable adapters to finetune lightweight neural network layers on top of pretrained weights. |
| Outcome: | The proposed adapters achieve on-par performances with the standard adapter architecture while using a considerably smaller number of adapter layers. |
AdapterHub Playground: Simple and Flexible Few-Shot Learning with Adapters (2022.acl-demo)
Copied to clipboard
Tilman Beck, Bela Bohlender, Christina Viehmann, Vincent Hane, Yanik Adamson, Jaber Khuri, Jonas Brossmann, Jonas Pfeiffer, Iryna Gurevych
| Challenge: | AdapterHub Playground is an open-access tool for researchers to use pretrained language models without writing a single line of code. |
| Approach: | They propose a tool which allows researchers to leverage pretrained models without writing a single line of code for a variety of NLP tasks. |
| Outcome: | The proposed model can be used for prediction, training and analysis of textual data without writing a single line of code. |
Lightweight Adapter Tuning for Multilingual Speech Translation (2021.acl-short)
Copied to clipboard
| Challenge: | Adapter tuning is an efficient alternative to fine-tuning in NLP . a multilingual model could be outperformed by its bilingual counterparts . |
| Approach: | They propose to use adapter tuning to optimize for multilingual speech translation . they use pre-trained models to freeze pre-train parameters and inject lightweight modules . |
| Outcome: | The proposed adapters can specialize to specific language pairs with low extra cost . the proposed models outperform bilingual models on high-resource language pairs . |
Modular and Parameter-Efficient Fine-Tuning for NLP Models (2022.emnlp-tutorials)
Copied to clipboard
| Challenge: | State-of-the-art language models in NLP perform best when fine-tuned even on small datasets. |
| Approach: | They provide an overview of parameter-efficient fine-tuning methods and highlight similarities and differences . they highlight benefits and usage scenarios of a neglected property of parameter efficient models . |
| Outcome: | This paper provides an overview of parameter-efficient fine-tuning methods . it highlights similarities and differences by presenting them in a unified view . |
Empowering parameter-efficient transfer learning by recognizing the kernel structure in self-attention (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods to fine-tune pre-trained language models are parameter efficient . fine- tuning the models requires multiple copies of the parameters, which is inefficient. |
| Approach: | They propose to use kernel-based adapters to tune only a few parameters while freezing the rest of the parameters. |
| Outcome: | The proposed methods achieve or improve strong performance over a diverse set of natural language generation and understanding tasks. |
On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation (2021.acl-long)
Copied to clipboard
Ruidan He, Linlin Liu, Hai Ye, Qingyu Tan, Bosheng Ding, Liying Cheng, Jiawei Low, Lidong Bing, Luo Si
| Challenge: | Existing studies have shown that adapter-based tuning is more parameter-efficient than fine-tuning. |
| Approach: | They propose to add adapter modules to a pretrained language model and update the parameters of adapter module when learning on a downstream task. |
| Outcome: | The proposed method outperforms fine-tuning on low-resource and cross-lingual tasks and settings. |
AdapterBias: Parameter-efficient Token-dependent Representation Shift for Adapters in NLP Tasks (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing approaches to train transformers with millions of parameters require large storage. |
| Approach: | They propose a transformer-based adapter architecture that adds a token-dependent shift to the hidden output of transformer layers to adapt to downstream tasks with only a vector and a linear layer. |
| Outcome: | The proposed model significantly reduces trainable parameters with minimal performance loss compared to fine-tuned models. |
Multilingual Machine Translation with Hyper-Adapters (2022.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual machine translation suffers from negative interference across languages. |
| Approach: | They propose a rescaling fix that reduces the number of parameters and enables training larger hyper-networks. |
| Outcome: | The proposed approach outperforms regular adapters and achieves the same performance with 12 times less parameters. |