Sparse coding for multitask and transfer learning

By Andreas Maurer, Massimiliano Pontil, Bernardino Romera-Paredes. 2012

195 citations recorded by the source corpus.

We investigate the use of sparse coding and dictionary learning in the context of multitask and transfer learning. The central assumption of our learning method is that the tasks parameters are well approximated by sparse linear combinations of the atoms of a dictionary on a high or infinite dimensional space. This assumption, together with the large quantity of available data in the multitask and transfer learning settings, allows a principled choice of the dictionary. We provide bounds on the generalization error of this approach, for both settings. Numerical experiments on one synthetic and two real datasets show the advantage of our method over single task learning, a previous method based on orthogonal and dense representation of the tasks and a related method learning task grouping.

Subfields: Artificial Intelligence, Computational Mechanics, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Representation & Self-Supervised Learning

Citations in the corpus

An Overview of Multi-task Learning (2018, methodology): “Unlike multi-layer feedforward neural networks, which are based on neural networks, the multitask feature learning (MTFL) method [5,6] and the multi-task sparse coding (MTSC) method [7] are formulated under the regularization framework by first transforming data instances as x̂j = Uxj and then learn”
A Survey on Multi-Task Learning (2017, methodology): “[14] analyze its generalization bound with the use of the Rademacher complexity.”
Continual learning: a feature extraction formalization, an efficient algorithm, and fundamental obstructions (2022, background): “Some recent theoretical works [18, 19, 26, 36, 20, 35, 9] provide generalization and sample complexity guarantees for certain formalizations of multi-task learning based on the existence of a good representation.”
A PAC-Bayesian bound for Lifelong Learning (2013, background): “However, except for a few cases, such as (Maurer, 2009; Maurer et al., 2013), their theoretical justifications are so far not well understood.”
Multi-Task Learning and Algorithmic Stability (2015, methodology): “The generalization bound for the multi-task sparse coding is analyzed in (Maurer, Pontil, and Romera-Paredes 2013; Maurer, Pontil, and Romera-Paredes 2014).”
Scalable Multitask Representation Learning for Scene Classification (2014, background): “There is also a connection between multitask learning [13] and supervised dictionary learning [12].”
Learning to Transfer: Transferring Latent Task Structures and Its Application to Person-Specific Facial Action Unit Detection (2015, background): “Recently, some works have aimed at automatically estimating the structure of relations among the different tasks [11, 13, 29].”
Lifelong Learning with Non-i.i.d. Tasks (2015, background): “Specific cases that were studied include feature learning [8] and sparse coding [9].”
Multi-Agent Distributed Lifelong Learning for Collective Knowledge Acquisition (2017, methodology): “This factorization has been analytically shown to enable knowledge transfer when dealing with related tasks by grouping the similar tasks [15, 22].”
Efficient Output Kernel Learning for Multiple Tasks (2015, background): “The advantage of MTL over learning tasks independently has been shown theoretically as well as empirically [1, 2, 3, 4, 5, 6, 7].”
Exact Recovery of Sparsely Used Overcomplete Dictionaries (2013, background): “Recent works [31, 28, 26] provide generalization bounds for predictive sparse coding, where the goal of the learned sparse representation is to obtain good performance on some predictive task.”
Continual Learning with Filter Atom Swapping (2022, methodology): “We are motivated by the literature on task subspace modeling (Evgeniou & Pontil, 2007; Maurer et al., 2013; Zhang & Yang, 2021; Romera-Paredes et al., 2013; Kumar & Daume III, 2012), where it is commonly assumed that task parameters lie in a low dimensional subspace, so that tasks can be modeled as…”
Understanding How Feature Structure Transfers in Transfer Learning (2017, methodology): “The notion of expected transfer risk [Maurer et al., 2013] associated with the learning algorithm AD is defined as follows: RE(AD) = El∼EESl∼µnll Ez∼µl`(y, hD,Sl(x)).”
Lifelong Learning with Weighted Majority Votes (2016, background): “These settings were theoretically analyzed in a series of works [18, 19, 23, 21] that have demonstrated that information transfer can lead to provable sample complexity reductions compared to solving each task independently.”