Deep Multi-task Representation Learning: A Tensor Factorisation Approach

By Yongxin Yang, Timothy M. Hospedales. 2016

271 citations recorded by the source corpus.

Most contemporary multi-task learning methods assume linear models. This setting is considered shallow in the era of deep learning. In this paper, we present a new deep multi-task representation learning framework that learns cross-task sharing structure at every layer in a deep network. Our approach is based on generalising the matrix factorisation techniques explicitly or implicitly used by many conventional MTL algorithms to tensor factorisation, to realise automatic learning of end-to-end knowledge sharing in deep networks. This is in contrast to existing deep learning approaches that need a user-defined multi-task sharing strategy. Our approach applies to both homogeneous and heterogeneous MTL. Experiments demonstrate the efficacy of our deep multi-task representation learning in terms of both higher accuracy and fewer design choices.

Subfields: Computational Mathematics, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mathematics

In Representation & Self-Supervised Learning

Citations in the corpus

An Overview of Multi-Task Learning in Deep Neural Networks (2017, methodology): “…learning [Kendall et al., 2017] 6.7 Tensor factorisation for MTL More recent work seeks to generalize existing approaches to MTL to Deep Learning: [Yang and Hospedales, 2017a] generalize some of the previously discussed matrix factorisation approaches using tensor factorisation to split the model…”
A Survey on Multi-Task Learning (2017, methodology): “product of several low-rank matrices or tensors [71].”
Scalable Recollections for Continual Lifelong Learning (2017, methodology): “We follow 90% multi-task training and 10% testing splits for Omniglot established in (Yang & Hospedales, 2017).”
Efficient Parametrization of Multi-domain Deep Neural Networks (2018, methodology): “Another MTL approach [1, 32, 20] focuses on dynamically generating DNN weights given the task identity.”
Deeper, Broader and Artier Domain Generalization (2017, positive): “In our case we exploit this idea both for compression – but across rather than within domains [13], as well as for crossdomain (rather than cross-category [31]) knowledge sharing.”
Meta-Learning in Neural Networks: A Survey (2020, background): “sentation [65], [66], as well as compute/memory savings.”
Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts (2018, methodology): “In fact, many DNN-based multi-task learning models are sensitive to factors such as the data distribution di erences and relationships among tasks [15, 34].”
Beyond Shared Hierarchies: Deep Multitask Learning through Soft Layer Ordering (2017, methodology): “Column-based approaches (Jou and Chang, 2016; Misra et al., 2016; Rusu et al., 2016; Yang and Hospedales, 2017), assign each task its own layer of task-speciﬁc parameters at each shared depth (Figure 1b).”
Learning Multiple Tasks with Multilinear Relationship Networks (2015, methodology): “We compare MRN with standard and state-of-the-art methods: Single-Task Learning (STL), MultiTask Feature Learning (MTFL) [2], Multi-Task Relationship Learning (MTRL) [31], Robust MultiTask Learning (RMTL) [5], and Deep Multi-Task Learning with Tensor Factorization (DMTL-TF) [27].”
Factorized Higher-Order CNNs With an Application to Spatio-Temporal Emotion Estimation (2019, background): “Beside theoretical study of the properties of deep neural networks [3], they have been especially studied in the context of reparametrizing existing layers [49].”
Domain Generalization: A Survey (2021, methodology): “Intuitively, MTL benefits from the effect of regularization brought by parameter sharing [129], which”
Multi-Task Learning with Deep Neural Networks: A Survey (2020, methodology): “Unfortunately, there is no empirical comparison of this tensor factorization approach with the other cross-talk architectures, and there hasn’t been much work extending the tensor factorization approach of (Yang and Hospedales, 2016a) for deep MTL.”
Collaborative Learning for Deep Neural Networks (2018, background): “Multi-task learning is an approach to learn multiple related tasks simultaneously so that knowledge obtained from each task can be reused by the others [4, 3, 21].”
TuckER: Tensor Factorization for Knowledge Graph Completion (2019, methodology): “Tucker decomposition, used widely in machine learning (Schein et al., 2016; Ben-Younes et al., 2017; Yang and Hospedales, 2017), factorizes a tensor into a core tensor multiplied by a matrix along each mode.”
Evolutionary architecture search for deep multitask networks (2018, background): “s architecture enablesthelearningoflayersthatareusedindifferentways at different depths for different tasks. Deep MTL has extended these ideas to domains where deep learning thrives, including vision [2, 17, 26, 31, 33, 35, 49, 51], speech [11,12,17,39,48],naturallanguageprocessing[4,6,9,17,24,27,50”
Trace Norm Regularised Deep Multi-Task Learning (2016, unknown): “…(‘conv1’) of size 5 × 5 × 3 × 32, (2) convolutional layer (‘conv2’) of size 3 × 3 × 32 × 64, (3) fully-connected layer (‘fc1’) of size 256 × 256, (4) fully-connected layer (‘fc2’ (1) ) of size 256 × 10 for the ﬁrst task and fully-connected layer (‘fc2’ (2) ) of size 256 × 20 for the second task.”
Tensor Based Knowledge Transfer Across Skill Categories for Robot Control (2017, methodology): “Multi-task Learning To jointly learn several tasks, we exploit ‘constructive’ multi-task learning [ Yang and Hospedales, 2015; 2017 ] .”