Towards More Efficient Post-training via Fourier Domain Adapter Framework

Challenge:	FDA reparameterizes the core projection operation of the adapter module directly in the Fourier domain.
Approach:	They propose a framework that reparameterizes the core projection operation of the adapter module directly in the Fourier domain.
Outcome:	The proposed framework outperforms existing parameter-efficient fine-tuning methods on GLUE, E2E NLG, and instruction tuning benchmarks.

Similar Papers

Overview

Challenge

Approach

Outcome

LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models (2023.emnlp-main)

Copied to clipboard

Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, Roy Lee

Language Models

Challenge:	Large language models (LLMs) have shown unprecedented performance across various tasks.
Approach:	They propose an easy-to-use framework that integrates adapters into LLMs . they evaluate adapters on 14 datasets from two different reasoning tasks .
Outcome:	The proposed framework can be used to fine-tune open-access language models with task-specific data and instruction data.

SQFT: Low-cost Model Adaptation in Low-precision Sparse Foundation Models (2024.findings-emnlp)

Copied to clipboard

Juan Munoz, Jinjie Yuan, Nilesh Jain

Language Models

Challenge:	Large pre-trained models are often adapted to a desired domain or task through a fine-tuning stage.
Approach:	They propose an end-to-end solution for sparse parameter-efficient fine-tuning of large pre-trained models.
Outcome:	The proposed approach can be used to combine sparse weights with low-rank adapters without losing sparsity and accuracy.

Unlocking Parameter-Efficient Fine-Tuning for Low-Resource Language Translation (2024.findings-naacl)

Copied to clipboard

Tong Su, Xin Peng, Sarubi Thillainathan, David Guzmán, Surangika Ranathunga, En-Shiun Lee

Responsible & Trustworthy NLP

Challenge:	Parameter-efficient fine-tuning (PEFT) methods are important in low-resource language (LRL) Neural Machine Translation (NMT) but their practical effectiveness varies significantly across different languages.
Approach:	They evaluated the performance of 8 parameters-efficient fine-tuning methods with 15 architectures using the SacreBLEU score.
Outcome:	The Houlsby+Inversion adapter outperforms the baseline architectures in both in-domain and out-domain tests and the Houlson+Inverter achieves the best performance overall.

On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation (2021.acl-long)

Copied to clipboard

Ruidan He, Linlin Liu, Hai Ye, Qingyu Tan, Bosheng Ding, Liying Cheng, Jiawei Low, Lidong Bing, Luo Si

Language Models

Challenge:	Existing studies have shown that adapter-based tuning is more parameter-efficient than fine-tuning.
Approach:	They propose to add adapter modules to a pretrained language model and update the parameters of adapter module when learning on a downstream task.
Outcome:	The proposed method outperforms fine-tuning on low-resource and cross-lingual tasks and settings.

Parameter-Efficient Language Model Tuning with Active Learning in Low-Resource Settings (2023.emnlp-main)

Copied to clipboard

Josip Jukić, Jan Snajder

Responsible & Trustworthy NLP

Challenge:	Pre-trained language models (PLMs) have ignited a surge in demand for effective fine-tuning techniques . data labeling is notoriously time-consuming and expensive, hindering the development of sizable labeled datasets .
Approach:	They propose to use active learning to reduce labeling costs by minimizing label complexity . they find PEFT adapter modules have significant potential in low-resource settings .
Outcome:	The proposed model outperforms FFT in low-resource settings and shows that it yields more stable representations of early and middle layers than FFT.

Parameter-Efficient Fine-Tuning without Introducing New Latency (2023.acl-long)

Copied to clipboard

Baohao Liao, Yan Meng, Christof Monz

Language Models

Challenge:	Parameter-efficient fine-tuning of pre-trained language models has been demonstrated to be effective, but its inherent characteristics limit its performance.
Approach:	They propose to generate a sparse mask in a task-agnostic manner by modifying only a small subset of existing parameters and adding new parameters.
Outcome:	The proposed method surpasses existing methods on the GLUE benchmark by a significant margin.

MEFT: Memory-Efficient Fine-Tuning through Sparse Adapter (2024.acl-long)

Copied to clipboard

Jitai Hao, Weiwei Sun, Xin Xin, Qi Meng, Zhumin Chen, Pengjie Ren, Zhaochun Ren

Semantic Text Processing

Challenge:	Parameter-Efficient Fine-tuning (PEFT) methods are limited on knowledge-intensive tasks due to the limited number of trainable parameters.
Approach:	They propose a mechanism that fine-tunes Large Language Models with larger adapters . they store and update the parameters of larger adapter adapters on the CPU .
Outcome:	The proposed method achieves comparable results to those obtained with larger memory capacities over the limited bandwidth of PCI Express (PCIe).

AlphaTuning: Quantization-Aware Parameter-Efficient Adaptation of Large-Scale Pre-Trained Language Models (2022.findings-emnlp)

Copied to clipboard

Se Jung Kwon, Jeonghoon Kim, Jeongin Bae, Kang Min Yoo, Jin-Hwa Kim, Baeseong Park, Byeongwook Kim, Jung-Woo Ha, Nako Sung, Dongsoo Lee

Language Models

Challenge:	Existing approaches to improve inference efficiency by accelerating model fine-tuning have not been thoroughly explored.
Approach:	They propose to combine parameter-efficient adaptation and model compression to accelerate model . they propose to freeze binary parameters and scale scaling factors for target tasks .
Outcome:	The proposed algorithm achieves >10x compression ratio under 4-bit quantization and >1,000x reduction in trainable parameters.

Mixture-of-Linguistic-Experts Adapters for Improving and Interpreting Pre-trained Language Models (2023.findings-emnlp)

Copied to clipboard

Raymond Li, Gabriel Murray, Giuseppe Carenini

Responsible & Trustworthy NLP

Challenge:	In recent years, pre-trained language models have become the de facto instrument for the field of natural language processing (NLP).
Approach:	They propose a method that injects linguistic structures into pre-trained language models in the parameter-efficient fine-tuning setting.
Outcome:	The proposed approach outperforms state-of-the-art methods with a comparable number of parameters.

Lightweight Adapter Tuning for Multilingual Speech Translation (2021.acl-short)

Copied to clipboard

Hang Le, Juan Pino, Changhan Wang, Jiatao Gu, Didier Schwab, Laurent Besacier

Green & Sustainable NLP

Challenge:	Adapter tuning is an efficient alternative to fine-tuning in NLP . a multilingual model could be outperformed by its bilingual counterparts .
Approach:	They propose to use adapter tuning to optimize for multilingual speech translation . they use pre-trained models to freeze pre-train parameters and inject lightweight modules .
Outcome:	The proposed adapters can specialize to specific language pairs with low extra cost . the proposed models outperform bilingual models on high-resource language pairs .