Tu Vu, Tong Wang, Tsendsuren Munkhdalai, Alessandro Sordoni, Adam Trischler, Andrew Mattarella-Micke, Subhransu Maji, Mohit Iyyer
| Challenge: | Recent advances in NLP demonstrate the effectiveness of training large-scale language models and transferring them to downstream tasks. |
| Approach: | They conduct an extensive study of the transferability between 33 NLP tasks across three broad classes of problems. |
| Outcome: | The proposed model can improve performance even with low-data source tasks that differ substantially from the target task. |
Similar Papers
Exploring the Role of Task Transferability in Large-Scale Multi-Task Learning (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent work has found that multi-task training with a large number of diverse tasks can uniformly improve downstream performance on unseen target tasks. |
| Approach: | They aim to disentangle the effect of scale and relatedness of tasks in multi-task representation learning by increasing the number of tasks and incorporating smaller sets of related tasks. |
| Outcome: | The proposed model improves on unseen target tasks by increasing the scale of multi-task learning to incorporate more tasks and developing similarity metrics to incorporate tasks related to the target task. |
Less is More: Parameter-Efficient Selection of Intermediate Tasks for Transfer Learning (2024.emnlp-main)
Copied to clipboard
| Challenge: | Prior methods producing useful task rankings are infeasible for large source pools . Embedding space maps (ESMs) reduce execution time and disk space usage . |
| Approach: | They introduce Embedded Space Maps (ESMs) that approximate the effect of fine-tuning a language model. |
| Outcome: | The proposed method reduces execution time and disk space usage by 10 and 278, respectively, while retaining high selection performance. |
Cross-Task Generalization Abilities of Large Language Models (2024.naacl-srw)
Copied to clipboard
| Challenge: | a thesis proposal advocates for the crucial role of cross-task generalization in NLP systems. |
| Approach: | They propose to benchmark cross-task generalization abilities with diverse NLP tasks . they also propose to develop model architectures for improving cross- task generalization . |
| Outcome: | This paper compares cross-task generalization abilities with diverse NLP tasks . it also analyzes and predicts the generalization landscape of current state-of-the-art large language models . |
Analyzing the Effect of Linguistic Similarity on Cross-Lingual Transfer: Tasks and Experimental Setups Matter (2025.findings-acl)
Copied to clipboard
| Challenge: | Prior work on cross-lingual transfer often focuses on a small set of languages from a few language families and/or a single task. |
| Approach: | They analyze cross-lingual transfer for 263 languages from a wide variety of language families . they include three popular NLP tasks: POS tagging, dependency parsing, topic classification . |
| Outcome: | The proposed approach is based on linguistic similarity measures for 263 languages . the results show that the effect of linguistic similarities on transfer performance depends on a range of factors . |
Transfer Learning in Natural Language Processing (N19-5)
Copied to clipboard
| Challenge: | supervised machine learning is based on learning in isolation, a single predictive model for a task using a dataset. |
| Approach: | They present an overview of modern transfer learning methods in natural language processing . they review examples and case studies on how models can be integrated and adapted . |
| Outcome: | The proposed methods improve upon the state-of-the-art on a wide range of NLP tasks. |
To Share or not to Share: Predicting Sets of Sources for Model Transfer Learning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to select transfer sources are limited by text and task similarity, which limits their application in transfer settings where both the task and the text domain change. |
| Approach: | They propose a model similarity measure that represents text and task similarity jointly to automatically determine which and how many sources to exploit. |
| Outcome: | The proposed approach improves performance by 24 F1 points for predicting promising sources across domains and tasks with similar models. |
Evidence > Intuition: Transferability Estimation for Encoder Selection (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on LM transferability have focused on a priori tuning of encoders . prior work has examined the different yet related tasks of performance prediction . |
| Approach: | They propose to generate quantitative evidence to predict which LM will perform best on a target task without fine-tuning all candidates. |
| Outcome: | The proposed model outperforms the standard of human practitioner ranking in 94% of the setups. |
Choose Your Transformer: Improved Transferability Estimation of Transformer Models on Classification Tasks (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing models for NLP tasks require fine-tuning, but it is computationally infeasible. |
| Approach: | They propose an approach that inexpensively estimates a ranking of the expected performance of a given set of transformer language models for a specific task. |
| Outcome: | The proposed model improves the Pearson correlation coefficient between the true model ranks and the estimate. |
Exploring the Effectiveness and Consistency of Task Selection in Intermediate-Task Transfer Learning (2024.acl-srw)
Copied to clipboard
| Challenge: | Identifying beneficial tasks to transfer from is a critical step toward successful intermediate-task transfer learning. |
| Approach: | They propose a method that measures pairwise token similarity using maximum inner product search to improve task prediction. |
| Outcome: | The proposed method improves task prediction scores from 2.59% to 3.96% for tasks requiring reasoning abilities, but not for reasoning abilities. |
How to Determine the Most Powerful Pre-trained Language Model without Brute Force Fine-tuning? An Empirical Survey (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Transferability estimation has been a topic of great interest in computer vision fields . a lack of a comprehensive comparison between these estimation methods is a problem . |
| Approach: | They conduct a thorough survey of existing methods to find the most suitable model . they also outline difficulties of consideration of training details and applicability to text generation . |
| Outcome: | The proposed methods perform well with superiorities in effectiveness and efficiency. |