Zhiheng Xi, Rui Zheng, Yuansen Zhang, Xuanjing Huang, Zhongyu Wei, Minlong Peng, Mingming Sun, Qi Zhang, Tao Gui
| Challenge: | Existing methods for predicting inter-task transferability are sparse and task-specific. |
| Approach: | They propose a method that uses connectivity patterns of neurons as a unique identifier associated with a task. |
| Outcome: | The proposed method outperforms baselines in predicting inter-task transferability across data regimes and transfer settings while keeping high efficiency in computation and storage. |
Similar Papers
Analysis and Prediction of NLP Models via Task Embeddings (2022.lrec-1)
Copied to clipboard
| Challenge: | Pretrained transformer-based encoders can be used to acquire rich text representations but need additional task supervision to be useful for downstream tasks. |
| Approach: | They propose a transformer to each MetaEval task and a neural network with a weighted encoder to perform the embeddings. |
| Outcome: | The proposed model outperforms baselines on GLUE tasks and can be used as a benchmark for future transfer learning research. |
Multi-Task Label Embedding for Text Classification (D18-1)
Copied to clipboard
| Challenge: | Existing work treats labels of each task as independent and meaningless one-hot vectors, which cause a loss of potential label information. |
| Approach: | They propose to combine multi-task learning with semantic vectors to convert labels into vectors . their results are based on extensive experiments on five benchmark datasets based in chinese . |
| Outcome: | The proposed model can improve performance on five benchmark datasets on text classification tasks. |
LITE: Intent-based Task Representation Learning Using Weak Supervision (2022.naacl-main)
Copied to clipboard
| Challenge: | To-do texts are often short and under-specified, which poses a challenge for current text representation models. |
| Approach: | They propose a neural multi-task learning framework that extracts representations of English to-do tasks with a multi-head attention mechanism on top of a pre-trained text encoder. |
| Outcome: | The proposed model outperforms baseline models on four downstream tasks and achieves error reduction of 38.7%. |
What’s in Your Embedding, And How It Predicts Task Performance (C18-1)
Copied to clipboard
| Challenge: | Attempts to find a single technique for general-purpose intrinsic evaluation of word embeddings have so far not been successful. |
| Approach: | They propose a method that quantifies interpretable characteristics of word vector neighborhoods and shows how they correlate with performance on 14 extrinsic and intrinsic task datasets. |
| Outcome: | The proposed approach enables multi-faceted evaluation, parameter search, and generally – a more principled, hypothesis-driven approach to development of distributional semantic representations. |
Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms (P18-1)
Copied to clipboard
Dinghan Shen, Guoyin Wang, Wenlin Wang, Martin Renqiang Min, Qinliang Su, Yizhe Zhang, Chunyuan Li, Ricardo Henao, Lawrence Carin
| Challenge: | Existing deep learning architectures to model compositionality in text sequences require a large number of parameters and expensive computations. |
| Approach: | They propose two additional pooling strategies over word embeddings for improved interpretability and hierarchical pooling for spatial (n-gram) information within text sequences. |
| Outcome: | The proposed pooling strategies improve interpretability and preserve spatial (n-gram) information within text sequences. |
Learning to Decompose and Organize Complex Tasks (2021.naacl-main)
Copied to clipboard
| Challenge: | Using a novel end-to-end pipeline, we propose a solution that consumes a complex task and induces 'dependency graphs' from unstructured text to represent sub-tasks and their relationships. |
| Approach: | They propose a pipeline that consumes a complex task and induces 'dependency graphs' from unstructured text to represent sub-tasks and their relationships. |
| Outcome: | The proposed pipeline outperforms state-of-the-art graph induction pipelines in a dataset of complex tasks with their sub-task graphs. |
Efficiently Tuned Parameters Are Task Embeddings (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for intermediate-task transfer are computationally infeasible to experiment with all intermediate combinations. |
| Approach: | They propose to use task-specific parameters updated in parameter-efficient tuning methods to predict inter-task transferability. |
| Outcome: | The proposed approach outperforms existing methods while being conceptually simple and computationally efficient. |
Exploring and Predicting Transferability across NLP Tasks (2020.emnlp-main)
Copied to clipboard
Tu Vu, Tong Wang, Tsendsuren Munkhdalai, Alessandro Sordoni, Adam Trischler, Andrew Mattarella-Micke, Subhransu Maji, Mohit Iyyer
| Challenge: | Recent advances in NLP demonstrate the effectiveness of training large-scale language models and transferring them to downstream tasks. |
| Approach: | They conduct an extensive study of the transferability between 33 NLP tasks across three broad classes of problems. |
| Outcome: | The proposed model can improve performance even with low-data source tasks that differ substantially from the target task. |
Towards Unified Task Embeddings Across Multiple Models: Bridging the Gap for Prompt-Based Large Language Models and Beyond (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing task embedding methods rely on fine-tuned, task-specific language models, which hinders their adaptability to prompt-guided Large Language Models (LLMs). |
| Approach: | They propose a framework for unified task embedding that harmonizes task embeds from various models within a single vector space. |
| Outcome: | The proposed framework harmonizes task embeddings from various models within a single vector space. |
Exploring the Effectiveness and Consistency of Task Selection in Intermediate-Task Transfer Learning (2024.acl-srw)
Copied to clipboard
| Challenge: | Identifying beneficial tasks to transfer from is a critical step toward successful intermediate-task transfer learning. |
| Approach: | They propose a method that measures pairwise token similarity using maximum inner product search to improve task prediction. |
| Outcome: | The proposed method improves task prediction scores from 2.59% to 3.96% for tasks requiring reasoning abilities, but not for reasoning abilities. |