The Microsoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understanding

Challenge:	MT-DNN is an open-source natural language understanding toolkit . it allows researchers and developers to train customized deep learning models .
Approach:	They present MT-DNN, an open-source natural language understanding toolkit . it is designed to facilitate rapid customization for a broad spectrum of NLU tasks . MT supports multi-task knowledge distillation, which can substantially compress a deep neural model without significant performance drop.
Outcome:	The proposed model can significantly compress a large model without significant performance drop.

Similar Papers

Overview

Challenge

Approach

Outcome

Multi-Task Deep Neural Networks for Natural Language Understanding (P19-1)

Copied to clipboard

Xiaodong Liu, Pengcheng He, Weizhu Chen, Jianfeng Gao

Language Models

Challenge:	Existing approaches to learning vector-space representations of text are multitask learning and language model pre-training.
Approach:	They propose a multi-task deep neural network (MT-DNN) that leverages cross-task data and incorporates a pre-trained bidirectional transformer language model.
Outcome:	The proposed model achieves state-of-the-art on ten NLU tasks and pushes the GLUE benchmark to 82.7% (2.2% absolute improvement)

AutoNLU: An On-demand Cloud-based Natural Language Understanding System for Enterprises (2020.aacl-demo)

Copied to clipboard

Nham Le, Tuan Lai, Trung Bui, Doo Soon Kim

Challenge:	AutoNLU is an on-demand cloud-based system that enables users to create and edit datasets and train and test different state-of-the-art NLU models.
Approach:	They introduce an on-demand cloud-based system that provides an easy-to-use interface . they build powerful keyphrase extraction models that achieve state-of-the-art results .
Outcome:	The proposed model achieves state-of-the-art on two public benchmarks and is easy to use and use.

NeuronBlocks: Building Your NLP DNN Models Like Playing Lego (D19-3)

Copied to clipboard

Ming Gong, Linjun Shou, Wutao Lin, Zhijie Sang, Quanjia Yan, Ze Yang, Feixiang Cheng, Daxin Jiang

Challenge:	Deep Neural Networks (DNN) have been widely employed in industry to address various natural language processing tasks.
Approach:	They propose an NLP toolkit that encapsulates neural network modules as building blocks to construct various DNN models with complex architecture.
Outcome:	The proposed toolkit can build, train, and test various DNN models with complex architecture.

A Learnable Skill Combination Strategy for Multi-task Learning in Natural Language Understanding (2026.findings-acl)

Copied to clipboard

Zhe Yang, Yi Huang, Yaqin Chen, Mengfei Guo, Xiaoting Wu, Junlan Feng

Language Models

Challenge:	a novel multi-task learning framework for domain-specific natural language understanding tasks addresses these limitations by combing multiple tasks into a single framework.
Approach:	They propose a multi-task learning framework that decomposes the language model into modular skill components and employs a dynamic, learnable skill-combination mechanism to adaptively handle diverse tasks.
Outcome:	The proposed framework surpasses conventional multi-task learning approaches in performance.

NeuralClassifier: An Open-source Neural Hierarchical Multi-label Text Classification Toolkit (P19-3)

Copied to clipboard

Liqun Liu, Funan Mu, Pengyu Li, Xin Mu, Jing Tang, Xingsheng Ai, Ran Fu, Lifeng Wang, Xing Zhou

Information Extraction & Text Mining

Challenge:	NeuralClassifier is a toolkit for hierarchical multi-label text classification.
Approach:	They propose a toolkit for neural hierarchical multi-label text classification . they use a variety of text encoders to implement the model .
Outcome:	The proposed model achieves comparable performance with reported results in the literature.

N-LTP: An Open-source Neural Language Technology Platform for Chinese (2021.emnlp-demo)

Copied to clipboard

Wanxiang Che, Yunlong Feng, Libo Qin, Ting Liu

Semantic Text Processing

Challenge:	Existing tools that teach an independent model for each task are not supported in Chinese.
Approach:	They propose an open-source neural language platform supporting six Chinese NLP tasks . source code, documentation, and pre-trained models are available at https://github.com/hit-SCIR/ltp .
Outcome:	The proposed platform supports six Chinese NLP tasks.

mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer (2021.naacl-main)

Copied to clipboard

Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, Colin Raffel

Text Generation

Challenge:	Current natural language processing pipelines often use transfer learning, where a model is pre-trained on a data-rich task before being fine-tuned on . this significantly limits their use given that roughly 80% of the world population does not speak English.
Approach:	They introduce a multilingual variant of T5 that was pre-trained on a new Common Crawl-based dataset covering 101 languages.
Outcome:	The proposed model achieves state-of-the-art on multilingual benchmarks and a simple technique to prevent accidental translation in the zero-shot setting.

Deep Learning for Natural Language Inference (N19-5)

Copied to clipboard

Samuel Bowman, Xiaodan Zhu

Reasoning

Challenge:	This tutorial discusses cutting-edge research on NLI, including recent advance on dataset development, cutting- edge deep learning models, and highlights from recent research on using NLI to understand capabilities and limits of deep learning for language understanding and reasoning.
Approach:	This tutorial discusses cutting-edge research on NLI, including recent advance on dataset development and cutting- edge deep learning models.
Outcome:	This tutorial discusses cutting-edge research on NLI, including recent advance on dataset development, cutting- edge deep learning models, and highlights from recent research on using NLI to understand capabilities and limits of deep learning model for language understanding and reasoning.

Deep Bayesian Natural Language Processing (P19-4)

Copied to clipboard

Jen-Tzung Chien

Information Extraction & Text Mining

Challenge:	Introduction to deep Bayesian learning for natural language addresses the fundamentals of statistical models and neural networks.
Approach:	This tutorial addresses the advances in deep Bayesian learning for natural language . it focuses on advanced Bayessian models and deep models . authors present case studies and domain applications to tackle different issues .
Outcome:	This tutorial focuses on advanced Bayesian models and deep models for natural language . case studies and domain applications are presented to tackle different issues in deep Bayessian processing, learning and understanding.

A Survey on Dynamic Neural Networks for Natural Language Processing (2023.findings-eacl)

Copied to clipboard

Canwen Xu, Julian McAuley

Challenge:	Dynamic neural networks can scale up pretrainable models with sub-linear increases in computation and time.
Approach:	They summarize the progress of three types of dynamic neural networks in NLP . skimming, mixtures of experts, and early exit are among the most popular .
Outcome:	The proposed models can scale up with sub-linear increases in computation and time . skimming, mixture of experts, and early exit are the most popular approaches .