The Microsoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understanding
By Xiaodong Liu, Yu Wang, Jianshu Ji, Hao Cheng, Xueyun Zhu, Emmanuel Awa, Pengcheng He, Weizhu Chen, Hoifung Poon, Guihong Cao and 1 more. 2020
62 citations recorded by the source corpus.
We present MT-DNN, an open-source natural language understanding (NLU) toolkit that makes it easy for researchers and developers to train customized deep learning models. Built upon PyTorch and Transformers, MT-DNN is designed to facilitate rapid customization for a broad spectrum of NLU tasks, using a variety of objectives (classification, regression, structured prediction) and text encoders (e.g., RNNs, BERT, RoBERTa, UniLM). A unique feature of MT-DNN is its built-in support for robust and transferable learning using the adversarial multi-task learning paradigm. To enable efficient production deployment, MT-DNN supports multi-task knowledge distillation, which can substantially compress a deep neural model without significant performance drop. We demonstrate the effectiveness of MT-DNN on a wide range of NLU applications across general and biomedical domains. The software and pre-trained models will be publicly available at https://github.com/namisan/mt-dnn.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Weizhu Chen
- Mohit Bansal
- Yejin Choi
- Percy Liang
- Xiaodong Liu
- Luke S. Zettlemoyer
- Pengcheng He
In Computer Vision and Pattern Recognition
- Jianfeng Gao
- Luc Van Gool
- Weizhu Chen
- Dacheng Tao
- Xiaogang Wang
- Mohit Bansal
- Trevor Darrell
- Xiaodong Liu
In Natural Language Processing
- Jianfeng Gao
- Weizhu Chen
- Xiaodong Liu
- Pengcheng He
- Yejin Choi
- Luke S. Zettlemoyer
- Mohit Bansal
- Percy Liang
Citations in the corpus
- Adversarial Training for Large Neural Language Models (2020, methodology): “For fine-tuning with or without adversarial training, we use the MT-DNN open-sourced toolkit (Liu et al., 2020, 2015) 6 .”
- Revisiting Few-sample BERT Fine-tuning (2020, background): “This change was introduced by Devlin et al. [7], and subsequently made its way into common open source libraries, including the official implementation,4 huggingface’s Transformers [50],5 AllenNLP [10], GluonNLP [12], jiant [47], MT-DNN [27], and FARM.6 As a result, this non-standard implementation ”
- SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization (2019, methodology): “Our implementation of SMART is based on BERT3 (Wolf et al., 2019), RoBERTa 4 (Liu et al., 2019c), MT-DNN 5 (Liu et al., 2020b) and HNN6.”
- A Robustly Optimized BERT Pre-training Approach with Post-training (2021, methodology): “Other approach is to optimize pre-training, like GP-T 2.0 (Radford et al., 2018), MT-DNN (Liu et al., 2020a), SpanBERT (Joshi et al., 2019), and AL-BERT (Lan et al., 2020).”
- DeepStruct: Pretraining of Language Models for Structure Prediction (2022, background): “In the context of NLP, weight sharing has been adopted in (Collobert and Weston, 2008; Yang et al., 2016; Liu et al., 2020).”
- Adversarial Training for Commonsense Inference (2020, methodology): “K-ADAPTER (Wang et al., 2020) further enhances RoBERTa word representations with multiple knowledge sources, such as factual knowledge obtained through Wikipedia and Wikidata and linguistic knowledge obtained through dependency parsing web texts.”