Papers with Adapters
Systematic Analysis for Pretrained Language Model Priming for Parameter-Efficient Fine-tuning (2024.naacl-srw)
Copied to clipboard
| Challenge: | Parameter-efficient (PE) methods for adapting pre-trained language models to downstream tasks are still lacking in many cases. |
| Approach: | They propose a general PE priming framework to enhance few-shot adaptation and generalization ability of PE methods. |
| Outcome: | The proposed framework reveals that the best priming strategy facilitates adaptation to target tasks. |
UKP-SQUARE: An Online Platform for Question Answering Research (2022.acl-demo)
Copied to clipboard
Tim Baumgärtner, Kexin Wang, Rachneet Sachdeva, Gregor Geigle, Max Eichler, Clifton Poth, Hannah Sterz, Haritz Puerto, Leonardo F. R. Ribeiro, Jonas Pfeiffer, Nils Reimers, Gözde Şahin, Iryna Gurevych
| Challenge: | Recent advances in NLP and information retrieval have given rise to a diverse set of question answering tasks that are of different formats and require different model architectures and setups. |
| Approach: | They propose an extensible online QA platform that allows users to query and analyze a large collection of modern Skills via a user-friendly web interface and integrated behavioural tests. |
| Outcome: | The proposed tool allows users to query and analyze a large collection of modern Skills via a user-friendly web interface and integrated behavioural tests. |
AdapterHub: A Framework for Adapting Transformers (2020.emnlp-demos)
Copied to clipboard
Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, Iryna Gurevych
| Challenge: | AdapterHub framework enables dynamic “stiching-in” of pre-trained adapters for different tasks and languages. |
| Approach: | They propose a framework that allows dynamic "stiching-in" of pre-trained adapters for different tasks and languages. |
| Outcome: | The proposed framework allows dynamic “stiching-in” of pre-trained adapters for different tasks and languages. |
Trankit: A Light-Weight Transformer-based Toolkit for Multilingual Natural Language Processing (2021.eacl-demos)
Copied to clipboard
| Challenge: | Trankit is a lightweight, pre-trained toolkit for multilingual natural language processing. |
| Approach: | They propose a transformer-based toolkit for multilingual natural language processing that trains pipelines over 100 languages and 90 pretrained pipelines for 56 languages. |
| Outcome: | The proposed tool outperforms existing pipelines over sentence segmentation, part-of-speech tagging, morphological feature tabbing, and dependency parsing while maintaining competitive performance over tokenization, multi-word token expansion, and lemmatization over 90 Universal Dependencies treebanks. |
Adapters: A Unified Library for Parameter-Efficient and Modular Transfer Learning (2023.emnlp-demo)
Copied to clipboard
Clifton Poth, Hannah Sterz, Indraneil Paul, Sukannya Purkayastha, Leon Engländer, Timo Imhof, Ivan Vulić, Sebastian Ruder, Iryna Gurevych, Jonas Pfeiffer
| Challenge: | Adapters is an open-source library that unifies parameter-efficient and modular transfer learning in large language models. |
| Approach: | They propose to integrate 10 different methods into a unified interface for parameter-efficient and modular transfer learning in large language models. |
| Outcome: | The proposed library is able to perform on multiple NLP tasks and is open-source. |
Cross-Lingual Transfer with Target Language-Ready Task Adapters (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing frameworks for (zero-shot) cross-lingual transfer employ separate language and task adapters which can be arbitrarily combined to perform any task to any target language. |
| Approach: | They propose to fine-tune 'target language-ready' adapters to the target language to achieve better transfer performance without sacrificing the modularity of MAD-X. |
| Outcome: | The proposed adapters outperform MAD-X and BAD-X on most tasks and languages while maintaining the modularity of MAD. |
AdaKron: An Adapter-based Parameter Efficient Model Tuning with Kronecker Product (2024.lrec-main)
Copied to clipboard
| Challenge: | Large Pretrained Language Models (PLMs) have billions of parameters, causing computational challenges to fine-tuning models. |
| Approach: | They propose an Adapter-based fine-tuning with the Kronecker product that combine the outputs of two small networks to form a final vector whose dimension is the product of the dimensions of the individual outputs. |
| Outcome: | The proposed method achieves the same performance levels as state-of-the-art methods on the GLUE benchmark . |
Zero-shot Cross-lingual Transfer With Learned Projections Using Unlabeled Target-Language Data (2023.acl-short)
Copied to clipboard
| Challenge: | Zero-shot cross-lingual transfer is enabled by pairing the language adapter in the target language with an appropriate task adapter within a source language. |
| Approach: | They propose to use unlabeled text to enhance zero-shot transfer by pairing language adapters with task adapters in a target language. |
| Outcome: | The proposed framework improves on three cross-lingual tasks with up to 11% relative improvement in Named Entity Recognition (NER), Question Answering (QA) and Natural Language Inference (NLI). |
Preserving Pre-trained Representation Space: On Effectiveness of Prefix-tuning for Large Multi-modal Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large multi-modal models (LMMs) are revolutionizing the way machines interact with the world, unlocking new possibilities across multi-dimensional applications. |
| Approach: | They propose a parameter-efficient fine-tuning strategy that combines both . they find that parameter tuning methods distort the feature representation space . |
| Outcome: | The proposed strategy preserves representation space while limiting performance on downstream tasks. |
Zero-Shot-BERT-Adapters: a Zero-Shot Pipeline for Unknown Intent Detection (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Intent discovery remains a crucial task in natural language processing . identifying novel, unseen intents remains one of the biggest challenges in this field . |
| Approach: | They propose a multi-language approach to intent discovery using Adapters and a Transformer architecture. |
| Outcome: | The proposed pipeline outperforms baselines in two zero-shot settings for intent classification and unseen intent discovery. |
Efficient Test Time Adapter Ensembling for Low-resource Language Varieties (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Specialized language and task adapters have been proposed to facilitate cross-lingual transfer of multilingual pretrained models. |
| Approach: | They propose a method that optimizes the ensemble weights of pretrained adapters for each test sentence by minimizing the entropy of its predictions. |
| Outcome: | The proposed method improves robustness to uncovered languages without training new adapters. |
Adaptation Approaches for Nearest Neighbor Language Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Semi-parametric Nearest Neighbor Language Models (kNN-LMs) have produced impressive gains over purely parametric LMs, however, there has been little investigation into adapting such models for new domains. |
| Approach: | They propose to adapt kNN-LMs to expand neighborhood retrieval over an additional adaptation datastore and adapt the weights of retrieved neighbors using a learned Rescorer module. |
| Outcome: | The proposed approach outperforms purely parametric adaptation and zero-shot models and achieves perplexity improvements of 17.1% and 16% across domains. |
Parameter-Efficient Neural Reranking for Cross-Lingual and Multilingual Retrieval (2022.coling-1)
Copied to clipboard
| Challenge: | State-of-the-art neural rankers are notoriously data-hungry and rarely used in multilingual and cross-lingual retrieval settings. |
| Approach: | They propose to use Sparse Fine-Tuning Masks and Adapters to transfer rankers trained on English data to other languages and cross-lingual setups by means of multilingual encoders. |
| Outcome: | The proposed methods outperform standard zero-shot transfer with full MMT fine-tuning while being more modular and reducing training times. |
Adapter Pruning using Tropical Characterization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on adapter pruning have not examined the optimal number of adapter parameters needed for downstream applications. |
| Approach: | They propose an adapter pruning approach that prunes adapter parameters without changing the orientation of underlying tropical hypersurfaces. |
| Outcome: | The proposed approach prunes adapter layers without changing the orientation of underlying tropical hypersurfaces. |
Composable Sparse Fine-Tuning for Cross-Lingual Transfer (2022.acl-long)
Copied to clipboard
| Challenge: | Adapters and sparse fine-tuning have been developed to improve transfer learning . a number of approaches have been proposed to improve performance of fine-untuners . |
| Approach: | They propose a method that fine-tunes the entire set of parameters of a large pretrained model . they use adapters and sparse fine-uning to improve model efficiency . |
| Outcome: | The proposed method outperforms adapters in cross-lingual transfer benchmarks. |
SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapters (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrain-finetuned models are increasingly complex and require more parameters to match the performance of full fine-tuning. |
| Approach: | They propose an efficient Adapter Tuning technique that freezes pretrained language models and fine-tunes a few extra modules. |
| Outcome: | The proposed setting outperforms the standard Adapter Tuning by 80% . the proposed setting is easy to use and has a high sparse ratio . |
Adaptable Adapters (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing work uses the same adapter architecture for every dataset regardless of the properties of the dataset or the amount of training data. |
| Approach: | They propose to use adaptable adapters to finetune lightweight neural network layers on top of pretrained weights. |
| Outcome: | The proposed adapters achieve on-par performances with the standard adapter architecture while using a considerably smaller number of adapter layers. |
HYPERTTS: Parameter Efficient Adaptation in Text to Speech Using Hypernetworks (2024.lrec-main)
Copied to clipboard
| Challenge: | Neural text-to-speech (TTS) systems limited to predefined speaker styles or specific sets of speaker IDs. |
| Approach: | They propose a network that can adapt adapter parameters to new speakers . they compare two domain adaptation settings and find it to be very efficient . |
| Outcome: | The proposed Adapters improve speech synthesis performance on two domains and compare them with baselines. |
Enhancing Scalability of Pre-trained Language Models via Efficient Parameter Sharing (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to scale pre-trained language models to a deeper model depth share all parameters or use extra blocks. |
| Approach: | They propose a parameter-efficient approach to scaling pre-trained language models to a deeper model depth using matrix product operator. |
| Outcome: | The proposed model scales pre-trained language models to a deeper model depth by 4x and achieves 0.1 points higher than BERT-large for GLUE score. |
LoNAS: Elastic Low-Rank Adapters for Efficient Large Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) reach hundreds of billions of parameters and require resources for training and inference stages. |
| Approach: | They propose a low-rank adapter to reduce the number of trainable parameters in a model and reduce memory requirements. |
| Outcome: | The proposed approach reduces memory and compute requirements while preserving performance. |
Meta-Adapter for Self-Supervised Speech Models: A Solution to Low-Resource Speech Recognition Challenges (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing self-supervised learning models can learn latent representations from large amounts of unlabeled data, but they are expensive to fine-tune. |
| Approach: | They develop a meta-adapter to obtain meta-initialized parameters for self-supervised models . meta-Adapters show better generalization and extensibility than traditional pretraining methods . |
| Outcome: | Experiments on common voice and FLEURS datasets show Meta-Adapter performs better on low-resource languages . authors show it can be used on 12 low-source languages, but it requires huge computational resources . |
GPT vs RETRO: Exploring the Intersection of Retrieval and Parameter-Efficient Fine-Tuning (2024.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained large language models can be used for specific tasks and unique information but lack the resources for extensive retraining. |
| Approach: | They propose to use PEFT methods to adapt large language models while minimizing compute requirements. |
| Outcome: | The proposed methods outperform GPT models in zero-shot settings but lag behind PEFT. |