The Benefit of Multitask Representation Learning

By Andreas Maurer, Massimiliano Pontil, Bernardino Romera-Paredes. 2015

422 citations recorded by the source corpus.

We discuss a general method to learn data representations from multiple tasks. We provide a justification for this method in both settings of multitask learning and learning-to-learn. The method is illustrated in detail in the special case of linear feature learning. Conditions on the theoretical advantage offered by multitask representation learning over independent task learning are established. In particular, focusing on the important example of half-space learning, we derive the regime in which multitask representation learning is beneficial over independent task learning, as a function of the sample size, the number of tasks and the intrinsic data dimensionality. Other potential applications of our results include multitask feature learning in reproducing kernel Hilbert spaces and multilayer, deep networks.

Subfields: Artificial Intelligence, Computational Mechanics, Mathematical Physics, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Mathematical Physics

In Representation & Self-Supervised Learning

Citations in the corpus

Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, methodology): “Multitask learning (Caruana, 1997) has been successfully applied to multiple domains, including NLP, speech processing, drug discovery and many others (Collobert and Weston, 2008; Deng et al., 2013; Ramsundar et al., 2015; Maurer et al., 2016; Ruder, 2017; Kaiser et al., 2017).”
The Ideal Continual Learner: An Agent That Never Forgets (2023, background): “This principled way of increasing the network width complements the empirical claim of Mirzadeh et al. (2022) and the experimental observation of Rusu et al. (2016); Yoon et al.”
Continual learning: a feature extraction formalization, an efficient algorithm, and fundamental obstructions (2022, background): “Some recent theoretical works [18, 19, 26, 36, 20, 35, 9] provide generalization and sample complexity guarantees for certain formalizations of multi-task learning based on the existence of a good representation.”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “For example, Maurer, Pontil, and Romera-Paredes (2016) extracted features for multiple tasks in a single low-dimensional shared representation.”
A Theoretical Analysis of Contrastive Unsupervised Representation Learning (2019, background): “are broadly useful on a distribution of tasks is done in multitask learning, specifically in the learning-to-learn model (Maurer et al., 2016) but using labeled data.”
Few-Shot Learning via Learning the Representation, Provably (2020, methodology): “Similar complexity measure also appeared in existing representation learning theory (Maurer et al., 2016).”
An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Models (2020, background): “It has been shown that MTL reduces the sample complexity of individual tasks compared to single-task learning (Caruana, 1997; Baxter, 2000; Maurer et al., 2016), thus may further improve the generalization capability of pre-trained models, especially on the minority groups.”
Bilevel Programming for Hyperparameter Optimization and Meta-Learning (2018, background): “Ideas from (Maurer et al., 2016; Denevi et al., 2018) may be useful in this direction.”
Provable Meta-Learning of Linear Representations (2020, methodology): “Second, in contrast to Maurer et al. (2016), we also provide guarantees for feature recovery in terms of the parameter estimation error—measured directly in the distance in the feature space.”
Risk Bounds for Transferring Representations With and Without Fine-Tuning (2017, methodology): “, 2016), and a distribution-dependent measure known as Gaussian complexity (Maurer et al., 2016).”
Heterogeneous Few-Shot Model Rectification With Semantic Mapping (2020, neutral): “tribution and are optimized simultaneously [42].”
Provable and Efficient Continual Representation Learning (2022, methodology): “Similar in spirit to this project, Maurer et al. (2016) provides generalization bounds for representation-based transfer learning in terms of the Rademacher complexities associated with the source and target tasks.”
Revisiting Meta-Learning as Supervised Learning (2020, background): “There are also theoretical analysis (Baxter, 2000; Maurer et al., 2016; Maurer, 2005).”
A Scaling Law for Syn2real Transfer: How Much Is Your Pre-training Effective? (2021, background): “Among others, some recent studies [16, 42, 62] derived an error bound of a fine-tuning task in the multi-task scenario based on complexity analysis; the bound takes an additive form O(An−1/2 +Bs−1/2), where n and s are the data size of pre-training and fine-tuning, respectively, with coefficients A ”
Meta-Learning by Adjusting Priors Based on Extended PAC-Bayes Theory (2017, methodology): “Finally, we compare methods which transfer knowledge from all of the training tasks: MLAP-M: The objective is based on Theorem 2 - the meta-learning bound obtained using Theorem 1 (McAllester’s single-task bound). MLAP-S: The objective is based on the meta-learning bound derived from Seeger’s single”
Sharing Knowledge in Multi-Task Deep Reinforcement Learning (2020, methodology): “Using Theorem 9 by Maurer et al. (2016), we can write: where L ( F ′ ) is the largest value for the Lipschitz constants in the function space F ′ , and D ( S ′′ ) is the Euclidean diameter of the set S ′′ .”