| Challenge: | Multi-task learning and self-training are two common ways to improve a machine learning model’s performance in settings with limited training data. |
| Approach: | They propose a transductive auxiliary task self-training procedure that trains a model on auxiliary tasks and test instances with auxiliary labels generated by a single-task version of the model. |
| Outcome: | The proposed method improves accuracy by 9.56% over the pure multi-task model for dependency relation tagging and 13.03% for semantic taging. |
Similar Papers
Efficient Multi-Task Auxiliary Learning: Selecting Auxiliary Data by Feature Similarity (2021.emnlp-main)
Copied to clipboard
| Challenge: | Multi-task auxiliary learning uses a set of relevant auxiliary tasks to improve performance of a primary task. |
| Approach: | They propose a time-efficient sampling method to select the most beneficial sub-datasets from the auxiliary tasks to achieve efficient multi-task auxiliary learning. |
| Outcome: | The proposed method significantly outperforms random sampling and ST-DNN on three benchmark datasets. |
Estimating the influence of auxiliary tasks for multi-task learning of sequence tagging tasks (2020.acl-main)
Copied to clipboard
| Challenge: | Multitask learning and transfer learning are techniques to overcome data scarcity . finding suitable auxiliary datasets for multitask learning is a trial-and-error approach . |
| Approach: | They propose to automatically assess the similarity of sequence tagging datasets to identify beneficial auxiliary data for MTL or TL setups. |
| Outcome: | The proposed methods can compute similarity between two sequence tagging datasets . they show that the same measures correlate with the change in test score of the auxiliary dataset . |
What can we learn from Semantic Tagging? (D18-1)
Copied to clipboard
| Challenge: | a recent study shows that multi-task learning improves performance of NLP tasks by exploiting similarities between tasks. |
| Approach: | They employ semantic tagging as an auxiliary task for three NLP tasks . they compare full neural network sharing, partial neural network shared and learning what to share . |
| Outcome: | The proposed model improves for part-of-speech tagging, universal dependency parsing and natural language inference. |
Multi-Task Learning for Sequence Tagging: An Empirical Study (C18-1)
Copied to clipboard
| Challenge: | Existing work on "pairwise" MTL has been validated in sequence tagging but key issues remain about its effectiveness. |
| Approach: | They propose three general multi-task learning approaches on 11 sequence tagging tasks. |
| Outcome: | The proposed approaches improve on 11 sequence tagging tasks. |
Friend-training: Learning from Models of Different but Related Tasks (2023.eacl-main)
Copied to clipboard
| Challenge: | Current self-training methods focus on improving model performance on a single task. |
| Approach: | They propose a cross-task self-training framework where models trained to do different tasks are used in iterative training, pseudo-labeling, and retraining processes to help each other for better selection of pseudo-labeled labels. |
| Outcome: | The proposed framework achieves the best performance compared to baselines on two dialogue understanding tasks. |
Transductive Learning of Neural Language Models for Syntactic and Semantic Analysis (D19-1)
Copied to clipboard
| Challenge: | despite its practical advantages, transductive learning is underexplored in natural language processing . despite the simplicity of the technique, it is understudied in natural languages . |
| Approach: | They conduct an empirical study of transductive learning for neural models . they fine-tune language models on an unlabeled test set to obtain test-set-specific word representations. |
| Outcome: | The proposed method improves state-of-the-art neural models in syntactic and semantic tasks. |
Multi-task Active Learning for Pre-trained Transformer-based Models (2022.tacl-1)
Copied to clipboard
| Challenge: | Multi-task learning requires annotating the same text with multiple annotation schemes, which can be costly and laborious. |
| Approach: | They propose to use multi-task active learning paradigm to optimize annotation processes by iteratively selecting unlabeled examples whose annotation is most valuable for the NLP model. |
| Outcome: | The proposed model minimizes annotation efforts for multi-task NLP models by iterating on the most valuable examples. |
Bayesian Learning for Neural Dependency Parsing (N19-1)
Copied to clipboard
| Challenge: | Several approaches for dependency parsing in the small data regime have been proposed. |
| Approach: | They propose to use stochastic gradient Langevin dynamics to generate samples from the approximated posterior to overcome the computational and statistical costs of the approximate inference step. |
| Outcome: | The proposed model outperforms the biaffine model on 6 languages with less than 5k training instances and improves across five languages. |
Multi-Task Learning for Cross-Lingual Abstractive Summarization (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing studies use pseudo cross-lingual abstractive summarization data to train neural encoder-decoders. |
| Approach: | They propose a multi-task learning framework for cross-lingual abstractive summarization that attaches a special token to the beginning of the input sentence to indicate the target task. |
| Outcome: | The proposed model achieves better performance than the model trained with only pseudo cross-lingual abstractive summarization data. |
Helping the Weak Makes You Strong: Simple Multi-Task Learning Improves Non-Autoregressive Translators (2022.emnlp-main)
Copied to clipboard
| Challenge: | Non-autoregressive (NAR) neural machine translation models require a conditional independence assumption on target sequences, resulting in less informative learning signals. |
| Approach: | They propose a model-agnostic multi-task learning framework to provide more informative learning signals for NAR models under conventional MLE training. |
| Outcome: | The proposed framework improves accuracy of multiple NAR baselines without additional decoding overhead. |