Representation Learning Using Multi-Task Deep Neural Networks for Semantic Classification and Information Retrieval

By Xiaodong Liu, Jianfeng Gao, Xiaodong He, Li Deng, Kevin Duh, Ye-Yi Wang. 2015

441 citations recorded by the source corpus.

Methods of deep neural networks (DNNs) have recently demonstrated superior performance on a number of natural language processing tasks. However, in most previous work, the models are learned based on either unsupervised objectives, which does not directly optimize the desired task, or singletask supervised objectives, which often suffer from insufficient training data. We develop a multi-task DNN for learning representations across multiple tasks, not only leveraging large amounts of cross-task data, but also benefiting from a regularization effect that leads to more general representations to help tasks in new domains. Our multi-task DNN approach combines tasks of multiple-domain classification (for query classification) and information retrieval (ranking for web search), and demonstrates significant gains over strong baselines in a comprehensive set of domain adaptation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Multi-task Sequence to Sequence Learning (2015, methodology): “In the context of deep neural networks, MTL has been applied successfully to various problems ranging from language (Liu et al., 2015), to vision (Donahue et al., 2014), and speech (Heigold et al., 2013; Huang et al., 2013).”
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “This is the approach advocated by the “MT-DNN” [Liu et al., 2015, 2019b].”
Adversarial Training for Large Neural Language Models (2020, methodology): “For ﬁne-tuning with or without adversarial training, we use the MT-DNN open-sourced toolkit (Liu et al., 2020, 2015) 6 .”
Story Ending Prediction by Transferable BERT (2019, methodology): “This work is also related to multi-task learning [Liu et al., 2015], where multiple tasks were jointly trained to get an overall performance improvement.”
Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding (2019, methodology): “We extend the knowledge distillation method (Hinton et al., 2015) to the multi-task learning setting (Caruana, 1997; Xu et al., 2018; Collobert et al., 2011; Zhang and Yang, 2017; Liu et al., 2015).”
Multi-Task Deep Neural Networks for Natural Language Understanding (2019, methodology): “To this end, we extend the MT-DNN model originally proposed in Liu et al. (2015) by incorporating BERT as its shared text encoding layers.”
A Hybrid Neural Network Model for Commonsense Reasoning (2019, methodology): “Therefore, inspired by multi-task learning (Caruana, 1997; Liu et al., 2015, 2019b), we pro-pose a hybrid neural network (HNN) model that combines the strengths of both neural language models and a semantic similarity model.”
A Knowledge-Grounded Neural Conversation Model (2017, background): “While (Liu et al., 2016) B LEU , which is known to better correlate with human judgments (Przybocki et al., 2008) including for response generation (Galley et al., 2015).”
Cross-Stitch Networks for Multi-task Learning (2016, background): “and their widespread applications in other fields, such as genomics [38], natural language processing [7, 8, 35] and computer vision [3, 10, 30, 31, 40, 51, 53, 58].”
Recurrent Neural Network for Text Classification with Multi-Task Learning (2016, methodology): “Neural networks based multi-task learning has proven effective in many NLP problems [Collobert and Weston, 2008; Liu et al. , 2015b ] .”
Relevance-based Word Embedding (2017, background): “ese models have been demonstrated to be eective in a number of IR tasks, including query expansion [11, 17, 40], query classication [23, 41], short text similarity [15], and document model estimation [2, 31].”
The Dialogue Dodecathlon: Open-Domain Knowledge and Image Grounded Conversational Agents (2019, methodology): “Multi-Task followed by Fine-Tuning As also performed in Liu et al. (2015); Raffel et al. (2019) we can try to train in a multi-task manner on all tasks, before ﬁne-tuning on a single task, and build a separate model performing this procedure for all tasks, in an attempt to improve single task…”
How to Fine-Tune BERT for Text Classification? (2019, methodology): “Liu et al. (2019) extend the MT-DNN model originally proposed in Liu et al. (2015) by incorporating BERT as its shared text encoding layers.”