Research on Task Discovery for Transfer Learning in Deep Neural Networks (2020.acl-srw)
Copied to clipboard
| Challenge: | Existing deep neural network based machine learning models suffer from overfitting and are sensitive to noise and examples that are not available in training data. |
| Approach: | They propose to use a novel multi-task learner to implement deep neural network based transfer learning models that can be used to improve generalization. |
| Outcome: | The proposed model performs better on two NLP tasks and is more efficient on other areas of machine learning, including Bioinformatics and Computer Vision. |
Similar Papers
Exploring and Predicting Transferability across NLP Tasks (2020.emnlp-main)
Copied to clipboard
Tu Vu, Tong Wang, Tsendsuren Munkhdalai, Alessandro Sordoni, Adam Trischler, Andrew Mattarella-Micke, Subhransu Maji, Mohit Iyyer
| Challenge: | Recent advances in NLP demonstrate the effectiveness of training large-scale language models and transferring them to downstream tasks. |
| Approach: | They conduct an extensive study of the transferability between 33 NLP tasks across three broad classes of problems. |
| Outcome: | The proposed model can improve performance even with low-data source tasks that differ substantially from the target task. |
Learning What to Share: Leaky Multi-Task Network for Text Classification (C18-1)
Copied to clipboard
| Challenge: | Existing approaches to multi-task learning suffer from the interference between tasks because they lack selection mechanism for feature sharing. |
| Approach: | They propose a multi-task convolutional neural network with the Leaky Unit which has memory and forgetting mechanism to filter the feature flows between tasks. |
| Outcome: | The proposed model can filter feature flows between tasks and improve performance on five datasets. |
Exploring the Role of Task Transferability in Large-Scale Multi-Task Learning (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent work has found that multi-task training with a large number of diverse tasks can uniformly improve downstream performance on unseen target tasks. |
| Approach: | They aim to disentangle the effect of scale and relatedness of tasks in multi-task representation learning by increasing the number of tasks and incorporating smaller sets of related tasks. |
| Outcome: | The proposed model improves on unseen target tasks by increasing the scale of multi-task learning to incorporate more tasks and developing similarity metrics to incorporate tasks related to the target task. |
Less is More: Parameter-Efficient Selection of Intermediate Tasks for Transfer Learning (2024.emnlp-main)
Copied to clipboard
| Challenge: | Prior methods producing useful task rankings are infeasible for large source pools . Embedding space maps (ESMs) reduce execution time and disk space usage . |
| Approach: | They introduce Embedded Space Maps (ESMs) that approximate the effect of fine-tuning a language model. |
| Outcome: | The proposed method reduces execution time and disk space usage by 10 and 278, respectively, while retaining high selection performance. |
Tied Multitask Learning for Neural Speech Translation (N18-1)
Copied to clipboard
| Challenge: | Recent efforts in endangered language documentation focus on collecting spoken language resources . BULB project uses mobile app to collect spoken resources accompanied by spoken translations . |
| Approach: | They propose a model where the second task decoder receives information from the first task . they apply regularization that encourages transitivity and invertibility to the model . |
| Outcome: | The proposed model improves performance on low-resource speech transcription and translation tasks. |
Transfer Learning in Natural Language Processing (N19-5)
Copied to clipboard
| Challenge: | supervised machine learning is based on learning in isolation, a single predictive model for a task using a dataset. |
| Approach: | They present an overview of modern transfer learning methods in natural language processing . they review examples and case studies on how models can be integrated and adapted . |
| Outcome: | The proposed methods improve upon the state-of-the-art on a wide range of NLP tasks. |
Towards Better Multi-task Learning: A Framework for Optimizing Dataset Combinations in Large Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Using a neural network, large language models can be trained on multiple tasks, allowing them to perform tasks efficiently. |
| Approach: | They propose a framework that leverages a neural network to select the best dataset combinations for enhancing multi-task learning (MTL) They propose to iteratively refine the selection, greatly improving efficiency while being model-, dataset-, and domain-independent. |
| Outcome: | The proposed framework iteratively refines the selection, greatly improving efficiency, while being model-, dataset-, and domain-independent. |
Searching for Effective Neural Extractive Summarization: What Works and What’s Next (P19-1)
Copied to clipboard
| Challenge: | Recent years have seen success in the use of deep neural networks on text summarization, but there is no clear understanding of why they perform so well or how they might be improved. |
| Approach: | They propose to use different types of model architectures to improve extractive summarization systems. |
| Outcome: | The proposed framework achieves state-of-the-art on CNN/DailyMail by a large margin based on observations and analysis. |
Multi-Task Retrieval for Knowledge-Intensive Tasks (2021.acl-long)
Copied to clipboard
Jean Maillard, Vladimir Karpukhin, Fabio Petroni, Wen-tau Yih, Barlas Oguz, Veselin Stoyanov, Gargi Ghosh
| Challenge: | Knowledge-intensive tasks require large amounts of knowledge about the world . recent neural retrieval models achieve better results by learning directly from task-specific training data. |
| Approach: | They propose a multi-task trained neural retrieval model that can be universally trained on a wide variety of problems. |
| Outcome: | The proposed model outperforms specialised retrievers on a few-shot setting and matches or improves state-of-the-art on multiple benchmarks. |
TaskWeb: Selecting Better Source Tasks for Multi-task NLP (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent work in NLP has shown that knowing task relationships via pairwise task transfer improves choosing one or more source tasks that help to learn a new target task. |
| Approach: | They propose a method to quantify task relationships via pairwise task transfer and build smaller training sets that improve zero-shot performances across 11 different target tasks. |
| Outcome: | The proposed method improves overall rankings and top-k precision of source tasks by 10% and 38%, respectively. |