| Challenge: | Experimental results show that fine-tuning pretrained language models on helpful intermediate tasks yields further gains. |
| Approach: | They propose to train an affinity scoring function to predict transferability between tasks by conditioning on task embeddings. |
| Outcome: | The proposed method efficiently identifies beneficial tasks for transfer learning. |
Similar Papers
Exploring the Effectiveness and Consistency of Task Selection in Intermediate-Task Transfer Learning (2024.acl-srw)
Copied to clipboard
| Challenge: | Identifying beneficial tasks to transfer from is a critical step toward successful intermediate-task transfer learning. |
| Approach: | They propose a method that measures pairwise token similarity using maximum inner product search to improve task prediction. |
| Outcome: | The proposed method improves task prediction scores from 2.59% to 3.96% for tasks requiring reasoning abilities, but not for reasoning abilities. |
Efficiently Tuned Parameters Are Task Embeddings (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for intermediate-task transfer are computationally infeasible to experiment with all intermediate combinations. |
| Approach: | They propose to use task-specific parameters updated in parameter-efficient tuning methods to predict inter-task transferability. |
| Outcome: | The proposed approach outperforms existing methods while being conceptually simple and computationally efficient. |
Exploring and Predicting Transferability across NLP Tasks (2020.emnlp-main)
Copied to clipboard
Tu Vu, Tong Wang, Tsendsuren Munkhdalai, Alessandro Sordoni, Adam Trischler, Andrew Mattarella-Micke, Subhransu Maji, Mohit Iyyer
| Challenge: | Recent advances in NLP demonstrate the effectiveness of training large-scale language models and transferring them to downstream tasks. |
| Approach: | They conduct an extensive study of the transferability between 33 NLP tasks across three broad classes of problems. |
| Outcome: | The proposed model can improve performance even with low-data source tasks that differ substantially from the target task. |
On Transferability of Prompt Tuning for Natural Language Processing (2022.naacl-main)
Copied to clipboard
Yusheng Su, Xiaozhi Wang, Yujia Qin, Chi-Min Chan, Yankai Lin, Huadong Wang, Kaiyue Wen, Zhiyuan Liu, Peng Li, Juanzi Li, Lei Hou, Maosong Sun, Jie Zhou
| Challenge: | Pre-trained language models (PLMs) can achieve comparable performance to full-parameter fine-tuning by tuning a few soft prompts, but require much more training time than fine-timing. |
| Approach: | They empirically investigate the transferability of soft prompts across different downstream tasks and PLMs to determine what decides prompt transferability. |
| Outcome: | The proposed method can achieve comparable performance to full-parameter fine-tuning by tuning a few soft prompts, but requires much more training time than fine-timing. |
Bayesian Multi-Task Transfer Learning for Soft Prompt Tuning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models have been fine-tuned for various NLP tasks . prompt tuning is a method that optimizes the output of the model to adapt to downstream tasks based on the posterior distribution of the source task. |
| Approach: | They propose a Bayesian approach to prompt tuning that optimizes for adapting pre-trained language models to downstream tasks rather than fine-tuning full model parameters. |
| Outcome: | The proposed approach outperforms the state-of-the-art methods on benchmark NLP tasks. |
SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer (2022.acl-long)
Copied to clipboard
| Challenge: | Recent studies show that pre-trained language models can be more efficient when they are larger than they are in their size. |
| Approach: | They propose a prompt-based transfer learning approach called SPoT: Soft Prompt Transfer that learns a soft prompt on one or more source tasks and initializes it for a target task. |
| Outcome: | The proposed approach outperforms Prompt Tuning and MODELTUNING on superGLUE benchmarks while using up to 27,000 fewer task-specific parameters. |
Less is More: Parameter-Efficient Selection of Intermediate Tasks for Transfer Learning (2024.emnlp-main)
Copied to clipboard
| Challenge: | Prior methods producing useful task rankings are infeasible for large source pools . Embedding space maps (ESMs) reduce execution time and disk space usage . |
| Approach: | They introduce Embedded Space Maps (ESMs) that approximate the effect of fine-tuning a language model. |
| Outcome: | The proposed method reduces execution time and disk space usage by 10 and 278, respectively, while retaining high selection performance. |
The Power of Scale for Parameter-Efficient Prompt Tuning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Unlike discrete text prompts used by GPT-3, soft prompts are learned through backpropagation and can be tuned to incorporate signals from any number of labeled examples. |
| Approach: | They propose a mechanism for learning "soft prompts" to condition frozen language models to perform specific downstream tasks. |
| Outcome: | The proposed method outperforms fewshot learning using GPT-3 and matches the quality of model tuning as models exceed billions of parameters. |
Parameter Efficient Multi-task Fine-tuning by Learning to Transfer Token-wise Prompts (2023.findings-emnlp)
Copied to clipboard
Muling Wu, Wenhao Liu, Jianhan Xu, Changze Lv, Zixuan Ling, Tianlong Li, Longtao Huang, Xiaoqing Zheng, Xuanjing Huang
| Challenge: | Prompt tuning has been proven to be successful on various tasks by incorporating a small number of trainable parameters while freezing large pre-trained language models. |
| Approach: | They propose a token-wise prompt tuning method that uses a bank of finer-grained soft prompt tokens to generate an instance-dependent prompt. |
| Outcome: | The proposed method performs far better than full parameter fine-tuned models and achieves state-of-the-art by tuning only 0.035% parameters on 14 datasets. |
Parameter-efficient Weight Ensembling Facilitates Task-level Knowledge Transfer (2023.acl-short)
Copied to clipboard
| Challenge: | Recent studies show that large pre-trained language models can be adapted to particular tasks in a parameter-efficient manner. |
| Approach: | They propose to use lightweight parameters to transfer them between tasks to obtain similarity between tasks. |
| Outcome: | The proposed methods show an improvement of 5%8% over baselines and could largely facilitate task-level knowledge transfer. |