Task-Driven Dictionary Learning

By Julien Mairal, Francis R. Bach, Jean Ponce. 2010

905 citations recorded by the source corpus.

Modeling data with linear combinations of a few elements from a learned dictionary has been the focus of much recent research in machine learning, neuroscience, and signal processing. For signals such as natural images that admit such sparse representations, it is now well established that these models are well suited to restoration tasks. In this context, learning the dictionary amounts to solving a large-scale matrix factorization problem, which can be done efficiently with classical optimization tools. The same approach has also been used for learning features from data for other purposes, e.g., image classification, but tuning the dictionary in a supervised way for these tasks has proven to be more difficult. In this paper, we present a general formulation for supervised dictionary learning adapted to a wide variety of tasks, and present an efficient algorithm for solving the corresponding optimization problem. Experiments on handwritten digit classification, digital art identification, nonlinear inverse image problems, and compressed sensing demonstrate that our approach is effective in large-scale settings, and is well suited to supervised and semi-supervised classification, as well as regression tasks for data that admit sparse representations.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

OptNet: Differentiable Optimization as a Layer in Neural Networks (2017, methodology): “, 2016) (the later with some inequality constraints) uses argmin differentiation within the context of a specific optimization problem, but doesn’t consider a particularly general setting; similarly, the older work of (Mairal et al., 2012) considered argmin differentiation for a LASSO problem, deriv”
Invariant Scattering Convolution Networks (2012, methodology): “If the training dataset is not augmented with deformations, the state of the art was achieved by deep-learning convolution networks [30], deformation models [17], [3], and dictionary learning [27].”
Deep Unfolding: Model-Based Inspiration of Novel Deep Architectures (2014, methodology): “Recently, discriminative methods have been applied to sparse dictionary based methods to achieve better performance in particular tasks [27].”
Super-Resolution with Deep Convolutional Sufficient Statistics (2015, background): “imization methods (Mairal et al. (2009)). Recent works have shown that better results can be obtained when the models are tuned on recovery performance rather than on data ﬁtting (Yang et al. (2012); Mairal et al. (2012)). However, the non-explicit form of the estimator leads to a more difﬁcult bi-l”
Fast and Robust Archetypal Analysis for Representation Learning (2014, methodology): “We ﬁrst show that it performs as well as sparse coding for learning codebooks of features in visual recognition tasks [24] and for signal classiﬁcation [14, 21, 23].”
Projective dictionary pair learning for pattern classification (2014, methodology): “As we introduced in Section 1, some DL methods [4, 5, 7] learn a shared dictionary for all classes and a classifier on the coding coefficients simultaneously, while some DL methods [8, 9, 10] learn a structured dictionary to promote discrimination between classes.”
Scalable Multitask Representation Learning for Scene Classification (2014, background): “There is also a connection between multitask learning [13] and supervised dictionary learning [12].”
A data driven method for feature transformation (2012, methodology): “Dictionary based image representations can also be improved through iterative subgradient methods[19] or through sparse coding [26, 22].”
New Era for Robust Speech Recognition (2017, methodology): “Using similar notation to [34], filter-and-sum enhancement can be written as follows:”
Structured sparsity through convex optimization (2011, background): “…non-convex global formulations (see, e.g., Zou, Hastie and Tibshirani, 2006; Moghaddam, Weiss and Avidan, 2006; Aharon, Elad and Bruckstein, 2006; Mairal et al., 2010) which estimate simultaneously all principal components, typically do not require the orthogonality of the components, and are…”
Discriminatively Activated Sparselets (2013, background): “…al., 2009; Zhu et al., 2010; Ott & Everingham, 2011; Girshick et al., 2011), (2) sparse coding and dictionary learning (Kreutz-Delgado et al., 2003; Mairal et al., 2009; 2012), and (3) modeling and learning with low-rank approximations (Freeman & Adelson, 1991; Manduchi et al., 1998; Wolf et al.,…”
Invariant Scattering Convolution Networks (2013, methodology): “If the training dataset is not augmented with deformations, the state of the art was achieved by deep-learning convolution networks [30], deformation models [17], [3], and dictionary learning [27].”
Decentralized Online Learning With Kernels (2017, background): “Such global stability guarantees are in contrast to specialized results for multi-agent kernel learning [34], [35] and alternative distributed online nonlinear function estimation methods such as dictionary learning [10], [15], [36] or neural networks [14], which suffer from instability due to the n”
Top-down visual saliency via joint CRF and dictionary learning (2012, methodology): “Recent advances in machine learning enable us to train task-specific dictionaries in a supervised manner [18, 28, 17].”
Robust and Discriminative Self-Taught Learning (2013, methodology): “, 2009), digital art authentication (Mairal et al., 2010), and many others.”
Sparse Dictionary Learning for Edit Propagation of High-Resolution Images (2014, background): “Minimizing Equation (1) can be transformed into an iterative sparse coding problem [18, 19], which can be solved efficiently.”
Scattering Representations for Recognition (2013, methodology): “If the training dataset is not augmented with deformations, the state of the art was achieved by deep-learning convolutional networks [RHBL07], deformation models [KDGH07; AT07], and dictionary learning [MBP10].”