Trace Norm Regularised Deep Multi-Task Learning

By Yongxin Yang, Timothy M. Hospedales. 2016

230 citations recorded by the source corpus.

We propose a framework for training multiple neural networks simultaneously. The parameters from all models are regularised by the tensor trace norm, so that each neural network is encouraged to reuse others' parameters if possible -- this is the main motivation behind multi-task learning. In contrast to many deep multi-task learning models, we do not predefine a parameter sharing strategy by specifying which layers have tied parameters. Instead, our framework considers sharing for all shareable layers, and the sharing strategy is learned in a data-driven way.

Subfields: Artificial Intelligence, Computational Mathematics, Statistical and Nonlinear Physics, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mathematics

In Statistical and Nonlinear Physics

In Representation & Self-Supervised Learning

Citations in the corpus

An Overview of Multi-Task Learning in Deep Neural Networks (2017, methodology): “…learning [Kendall et al., 2017] 6.7 Tensor factorisation for MTL More recent work seeks to generalize existing approaches to MTL to Deep Learning: [Yang and Hospedales, 2017a] generalize some of the previously discussed matrix factorisation approaches using tensor factorisation to split the model…”
A Survey on Multi-Task Learning (2017, methodology): “Based on such tensor representations, several tensor trace norms that are based on the matrix trace norm [49], are used in [52] as regularizers to identify the low-rank structure of the parameter tensor.”
Gradient Surgery for Multi-Task Learning (2020, methodology): “The multi-task formulation has been applied to many different settings, including supervised learning (Zhang et al., 2014; Long & Wang, 2015; Yang & Hospedales, 2016; Sener & Koltun, 2018; Zamir et al., 2018) and reinforcement-learning (Espeholt et al., 2018; Wilson et al., 2007), as well as many…”
Latent Multi-Task Architecture Learning (2017, methodology): “, as well as at what layers the network has learned the best representations of the input sequences. Matrix Regularization We cast learning what to share as a matrix regularization problem, following [15, 29]. Assume Mdifferent tasks that are loosely related, with Mpotentially non-overlapping datase”
Sluice networks: Learning what to share between loosely related tasks (2017, methodology): “We cast learning what to share as a matrix regularization problem, following [15, 29].”
Which Tasks Should Be Learned Together in Multi-task Learning? (2019, background): “Examples include [9, 11, 26, 41, 43, 20] The canonical example of soft parameter sharing can be seen in [11].”
Multi-Task Learning as Multi-Objective Optimization (2018, methodology): “…gradient-based optimization, following the success of deep MTL in computer vision (Bilen and Vedaldi, 2016; Misra et al., 2016; Rudd et al., 2016; Yang and Hospedales, 2016; Kokkinos, 2017; Zamir et al., 2018), natural language processing (Collobert and Weston, 2008; Dong et al., 2015; Liu et…”
Learning what to share between loosely related tasks (2017, methodology): “We cast learning what to share as a matrix regularization problem, following (Jacob et al., 2009; Yang and Hospedales, 2017).”
Visual Reasoning by Progressive Module Networks (2018, background): “The dominant approach to multi-task learning is to have a model that shares parameters in a soft (Duong et al., 2015; Yang & Hospedales, 2017) or hard way (Caruana, 1993).”
Efficiently Identifying Task Groupings for Multi-Task Learning (2021, background): “A plethora of multi-task methods addressing what parameters to share among tasks in a model have been developed, such as Neural Architecture Search [19, 47, 49, 35, 22, 39], Soft-Parameter Sharing [40, 14, 52], and asymmetric information transfer [31, 44, 32] to improve multi-task performance.”
Multi-Task Learning with Deep Neural Networks: A Survey (2020, methodology): “Unfortunately, there is no empirical comparison of this tensor factorization approach with the other cross-talk architectures, and there hasn’t been much work extending the tensor factorization approach of (Yang and Hospedales, 2016a) for deep MTL.”
Multi-task Learning (2020, methodology): “…been developed, such as Neural Architecture Search (Guo et al., 2020; Sun et al., 2019; Vandenhende et al., 2019; Rusu et al., 2016; Huang et al., 2018; Lu et al., 2017) and Soft-Parameter Sharing (Misra et al., 2016; Duong et al., 2015; Yang & Hospedales, 2016), to improve multi-task performance.”