Papers with MT-DNN
The Microsoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understanding (2020.acl-demos)
Copied to clipboard
Xiaodong Liu, Yu Wang, Jianshu Ji, Hao Cheng, Xueyun Zhu, Emmanuel Awa, Pengcheng He, Weizhu Chen, Hoifung Poon, Guihong Cao, Jianfeng Gao
| Challenge: | MT-DNN is an open-source natural language understanding toolkit . it allows researchers and developers to train customized deep learning models . |
| Approach: | They present MT-DNN, an open-source natural language understanding toolkit . it is designed to facilitate rapid customization for a broad spectrum of NLU tasks . MT supports multi-task knowledge distillation, which can substantially compress a deep neural model without significant performance drop. |
| Outcome: | The proposed model can significantly compress a large model without significant performance drop. |
Efficient Multi-Task Auxiliary Learning: Selecting Auxiliary Data by Feature Similarity (2021.emnlp-main)
Copied to clipboard
| Challenge: | Multi-task auxiliary learning uses a set of relevant auxiliary tasks to improve performance of a primary task. |
| Approach: | They propose a time-efficient sampling method to select the most beneficial sub-datasets from the auxiliary tasks to achieve efficient multi-task auxiliary learning. |
| Outcome: | The proposed method significantly outperforms random sampling and ST-DNN on three benchmark datasets. |
LIMIT-BERT : Linguistics Informed Multi-Task BERT (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing language models are usually trained on large amounts of unlabeled text data. |
| Approach: | They propose a multi-task language representations learning framework for multi-linguistics tasks by Multi-Task Learning. |
| Outcome: | The proposed model outperforms the baseline Whole Word Masking BERT on both dependency and constituent syntactic/semantic parsing, GLUE benchmark, and SNLI task. |
Multi-Task Deep Neural Networks for Natural Language Understanding (P19-1)
Copied to clipboard
| Challenge: | Existing approaches to learning vector-space representations of text are multitask learning and language model pre-training. |
| Approach: | They propose a multi-task deep neural network (MT-DNN) that leverages cross-task data and incorporates a pre-trained bidirectional transformer language model. |
| Outcome: | The proposed model achieves state-of-the-art on ten NLU tasks and pushes the GLUE benchmark to 82.7% (2.2% absolute improvement) |
GradTS: A Gradient-Based Automatic Auxiliary Task Selection Method Based on Transformer Networks (2021.emnlp-main)
Copied to clipboard
| Challenge: | A key problem in multi-task learning (MTL) research is how to select high-quality auxiliary tasks automatically. |
| Approach: | They propose an automatic auxiliary task selection method based on gradient calculation in Transformer-based models that improves MT-DNN performance. |
| Outcome: | The proposed method improves MT-DNN performance on 8 natural language understanding (GLUE) tasks, while costing less than AUTOSEM and comparable GPU consumption. |