| Challenge: | Existing methods for intermediate-task transfer are computationally infeasible to experiment with all intermediate combinations. |
| Approach: | They propose to use task-specific parameters updated in parameter-efficient tuning methods to predict inter-task transferability. |
| Outcome: | The proposed approach outperforms existing methods while being conceptually simple and computationally efficient. |
Similar Papers
Learning to Predict Task Transferability via Soft Prompt (2023.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that fine-tuning pretrained language models on helpful intermediate tasks yields further gains. |
| Approach: | They propose to train an affinity scoring function to predict transferability between tasks by conditioning on task embeddings. |
| Outcome: | The proposed method efficiently identifies beneficial tasks for transfer learning. |
What to Pre-Train on? Efficient Intermediate Task Selection (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning intermediate tasks are inefficient and expensive. |
| Approach: | They propose to use a set of 42 intermediate and 11 target English classification, multiple choice, question answering, and sequence tagging tasks to identify the best settings for intermediate transfer learning. |
| Outcome: | The proposed methods achieve an average Regret@3 of 1% across all target tasks. |
Exploring the Effectiveness and Consistency of Task Selection in Intermediate-Task Transfer Learning (2024.acl-srw)
Copied to clipboard
| Challenge: | Identifying beneficial tasks to transfer from is a critical step toward successful intermediate-task transfer learning. |
| Approach: | They propose a method that measures pairwise token similarity using maximum inner product search to improve task prediction. |
| Outcome: | The proposed method improves task prediction scores from 2.59% to 3.96% for tasks requiring reasoning abilities, but not for reasoning abilities. |
Less is More: Parameter-Efficient Selection of Intermediate Tasks for Transfer Learning (2024.emnlp-main)
Copied to clipboard
| Challenge: | Prior methods producing useful task rankings are infeasible for large source pools . Embedding space maps (ESMs) reduce execution time and disk space usage . |
| Approach: | They introduce Embedded Space Maps (ESMs) that approximate the effect of fine-tuning a language model. |
| Outcome: | The proposed method reduces execution time and disk space usage by 10 and 278, respectively, while retaining high selection performance. |
Exploring and Predicting Transferability across NLP Tasks (2020.emnlp-main)
Copied to clipboard
Tu Vu, Tong Wang, Tsendsuren Munkhdalai, Alessandro Sordoni, Adam Trischler, Andrew Mattarella-Micke, Subhransu Maji, Mohit Iyyer
| Challenge: | Recent advances in NLP demonstrate the effectiveness of training large-scale language models and transferring them to downstream tasks. |
| Approach: | They conduct an extensive study of the transferability between 33 NLP tasks across three broad classes of problems. |
| Outcome: | The proposed model can improve performance even with low-data source tasks that differ substantially from the target task. |
Parameter-efficient Weight Ensembling Facilitates Task-level Knowledge Transfer (2023.acl-short)
Copied to clipboard
| Challenge: | Recent studies show that large pre-trained language models can be adapted to particular tasks in a parameter-efficient manner. |
| Approach: | They propose to use lightweight parameters to transfer them between tasks to obtain similarity between tasks. |
| Outcome: | The proposed methods show an improvement of 5%8% over baselines and could largely facilitate task-level knowledge transfer. |
Hit the Nail on the Head: Parameter-Efficient Multi-task Tuning via Human Language Intervention (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that PEFT on small pre-trained language models improves multitasking capabilities. |
| Approach: | They propose a multi-task learning framework that enables transfer of prior knowledge across tasks . they attach task descriptions to input samples and map them to task embeddings . |
| Outcome: | The proposed method improves performance on a T5 model and in decoder-only models . |
Divergence-Based Domain Transferability for Zero-Shot Classification (2023.findings-eacl)
Copied to clipboard
| Challenge: | a recent study shows that fine-tuning of neural models can improve performance on language-based tasks without brute-force searching effective task combinations. |
| Approach: | They propose to use divergence measures to estimate whether one task pair will perform better than another . they use 58 tasks and 6,600 task pair combinations to study the effect of different tuning methods . |
| Outcome: | The proposed method reduces end-to-end runtime by 40% by estimating transferability . the proposed method is based on 58 tasks and over 6,600 task pair combinations . |
Making Parameter-efficient Tuning More Efficient: A Unified Framework for Classification Tasks (2022.coling-1)
Copied to clipboard
Xin Zhou, Ruotian Ma, Yicheng Zou, Xuanting Chen, Tao Gui, Qi Zhang, Xuanjing Huang, Rui Xie, Wei Wu
| Challenge: | Large pre-trained language models (PLMs) have demonstrated superior performance in industrial applications. |
| Approach: | They propose a framework that re-uses existing parameter-efficient methods with a unified classifier. |
| Outcome: | The proposed framework improves the efficiency of existing parameter-efficient methods with a unified classifier. |
ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Prompts (2022.emnlp-main)
Copied to clipboard
| Challenge: | a new multi-task, parameter-efficient language model tuning method learns to transfer knowledge across different tasks via a mixture of soft prompts. |
| Approach: | They propose a multi-task, parameter-efficient language model tuning method that uses soft prompts to learn to transfer knowledge across different tasks. |
| Outcome: | The proposed method outperforms prompt tuning and outperfies or matches fully fine-tuned tuning approaches that use 10 times more parameters. |