Learning Multiple Tasks with Deep Relationship Networks

By Mingsheng Long, Jianmin Wang. 2015

98 citations recorded by the source corpus.

Deep neural networks trained on large-scale dataset can learn transferable features that promote learning multiple tasks for inductive transfer and labeling mitigation. As deep features eventually transition from general to specific along the network, a fundamental problem is how to exploit the relationship structure across different tasks while accounting for the feature transferability in the task-specific layers. In this work, we propose a novel Deep Relationship Network (DRN) architecture for multi-task learning by discovering correlated tasks based on multiple task-specific layers of a deep convolutional neural network. DRN models the task relationship by imposing matrix normal priors over the network parameters of all task-specific layers, including higher feature layers and classifier layer that are not transferable safely. By jointly learning the transferable features and task relationships, DRN is able to alleviate the dilemma of negative-transfer in the feature layers and under-transfer in the classifier layer. Empirical evidence shows that DRN yields state-of-the-art classification results on standard multi-domain object recognition datasets.

Subfields: Artificial Intelligence, Computational Mathematics, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mathematics

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Optimization & Training

Citations in the corpus

An Overview of Multi-Task Learning in Deep Neural Networks (2017, background): “Figure 3: A Deep Relationship Network with shared convolutional and task-speciﬁc fully connected layers with matrix priors [Long and Wang, 2015]”
Active Long Term Memory Networks (2016, background): “The introduction of double streams architecture inspired by the classical Siamese Network [44] for metric learning, and the consequent generalization of the multi-task framework to domain adaptation [45, 46] offers a natural extension of KD, where distillation happens between streams made to handle ”
GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks (2017, methodology): “Deep networks have been used to solve various subsets of multiple vision tasks, from 3-task networks (Eigen & Fergus (2015); Teichmann et al. (2016)) to much larger subsets as in UberNet (Kokkinos (2016)).”
Multi-Task Learning as Multi-Objective Optimization (2018, methodology): “, 2007; Bakker and Heskes, 2003) or a joint dictionary (Argyriou et al., 2007; Long and Wang, 2015; Yang and Hospedales, 2016; Ruder, 2017).”
Recommending what video to watch next: a multitask ranking system (2019, background): “, natural language processing [12], and computer vision [27].”
Stochastic Filter Groups for Multi-Task CNNs: Learning Specialist and Generalist Convolution Kernels (2019, methodology): “These methods set out to learn a set a vectors that control which features are shared within a layer and how these are distributed across [19, 13, 4, 12].”
Branched Multi-Task Networks: Deciding what layers to share (2019, background): “Multilinear relationship networks [36] extend this general framework by placing tensor normal priors on the parameter set of the fully connected layers.”
Knowledge Distillation for Bilingual Dictionary Induction (2017, background): “Domain adaptation, for which there is a long history, is also related to our work (Ben-David et al., 2007; Daum´e III, 2007; Pan et al., 2010; Long and Wang, 2015).”
PAD-Net: Multi-tasks Guided Prediction-and-Distillation Network for Simultaneous Depth Estimation and Scene Parsing (2018, background): “In other words, it is a crucial point that how to design a good network architecture so that the network communicates or shares information based on the multi-modal data for different tasks, while other deep multi-task learning models such as Cross-stitch Net [38], Sluice Net [44], and Deep Relation”
Multi-task Learning based Pre-trained Language Model for Code Completion (2020, background): “e more general to new tasks in the future. Multi-task learning has been successfully used in many fields including natural language processing [10, 14, 31], speech recognition [9] and computer vision [32, 34]. 2.3 Pre-trained Language Models Language model pre-training has shown to be effective for ”
Knowledge Adaptation: Teaching to Adapt (2017, methodology): “…that maps source and target domain data to a space that encourages in-domain similarity, while Pan et al. (2010) proposed a spectral feature alignment algorithm to align domain-specific words into meaningful clusters, while Long and Wang (2015) use multi-task learning to avoid negative transfer.”
MultiNet: Real-time Joint Semantic Reasoning for Autonomous Driving (2016, background): “Several approaches have been proposed in the context of CNNs [30, 28] but applications have mainly been focussed on face recognition tasks [51, 47, 37].”