Papers with PEFTs
Layer-wise Importance Matters: Less Memory for Better Performance in Parameter-efficient Fine-tuning of Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Parameter-Efficient Fine-Tuning (PEFT) methods have gained popularity for adapting pre-trained Large Language Models (LLMs) to downstream tasks. |
| Approach: | They propose a method to optimize the importance of full layers with layer-wise importance scoring by leveraging the estimated importance scores. |
| Outcome: | The proposed method is compatible with PEFT methods that operate on a per-layer basis and achieves better performance. |
PEFTDebias : Capturing debiasing information using PEFTs (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent research suggests that biases acquired during pretraining can propagate to downstream models, resulting in superficial text dependencies and potential implicit bias. |
| Approach: | They propose a parameter-efficient fine-tuning approach to mitigate implicit biases within foundation models by incorporating parameters into the model and freezing them during the fine-uning process. |
| Outcome: | The proposed method reduces biases in foundation models by incorporating parameters and freezing them during fine-tuning. |
FedReFT: Federated Representation Fine-Tuning with All-But-Me Aggregation (2026.findings-eacl)
Copied to clipboard
| Challenge: | Representation Fine-Tuning (ReFT) adapts large pre-trained models by updating only a small subset of parameters. |
| Approach: | They propose a method that uses sparse intervention layers to steer hidden representations directly to capture rich semantic information. |
| Outcome: | The proposed approach outperforms PEFTs on commonsense reasoning, arithmetic reasoning, and GLUE benchmarks while maintaining a high parameter efficiency. |
When does Parameter-Efficient Transfer Learning Work for Machine Translation? (2022.emnlp-main)
Copied to clipboard
| Challenge: | Prior work indicates that parametric fine-tuning methods may not work as well for machine translation (MT). |
| Approach: | They propose to use parameter-efficient fine-tuning methods to adapt large pre-trained models while only tuning a small number of parameters. |
| Outcome: | The proposed methods outperform full fine-tuning for many downstream tasks when the parameter budget corresponds to 10% of the model parameters. |
Know-Adapter: Towards Knowledge-Aware Parameter-Efficient Transfer Learning for Few-shot Named Entity Recognition (2024.lrec-main)
Copied to clipboard
| Challenge: | Named entity recognition (NER) is a fundamental task in natural language processing. |
| Approach: | They propose a knowledgeable adapter to incorporate structure and semantic knowledge of knowledge graphs into PLMs for few-shot NER. |
| Outcome: | The proposed adapter improves the quality of retrieved information by adding explicit knowledge from external sources to PEFTs. |