Learning Fast Approximations of Sparse Coding

By Karol Gregor, Yann LeCun. 2010

2,065 citations recorded by the source corpus.

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

Learning to Optimize Neural Nets (2017, background): “A different line of work (Gregor & LeCun, 2010; Sprechmann et al., 2013) parameterizes intermediate operands of special-purpose solvers for a class of optimization problems that arise in sparse coding and learns them using supervised learning.”
Deformable part models are convolutional neural networks (2014, methodology): “The idea of unrolling (or “time unfolding”) an inference algorithm in order to construct a ﬁxed-depth network was explored by Gregor and LeCun in application to sparse coding [18].”
Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network (2016, background): “In Wang et al. [57] the authors encode a sparse representation prior into their feed-forward network architecture based on the learned iterative shrinkage and thresholding algorithm (LISTA) [22].”
Feature Purification: How Adversarial Training Performs Robust Deep Learning (2020, background): “There are many existing theoretical works studying algorithm for sparse coding [9, 17, 43, 47, 54, 68, 85, 91, 94], however, these algorithms share little similarity to training a neural network.”
Learning to Learn with Generative Models of Neural Network Checkpoints (2022, background): “They are typically trained with unrolled optimization (Hochreiter et al., 2001; Younger et al., 2001; Gregor & LeCun, 2010; Andrychowicz et al., 2016; Ravi & Larochelle, 2017; Wichrowska et al., 2017; Lv et al., 2017; Metz et al., 2019; 2022) or reinforcement learning (Li & Malik, 2016; 2017).”
Representation Learning: A Review and New Perspectives (2012, background): “Gregor and LeCun (2010a); Le et al. (2010) also demonstrated the technique of tiledconvolution, where parameters are shared only between feature extractors whose receptive fields are k steps away (so the ones looking at immediate neighbor locations are not shared).”
Stacked What-Where Auto-encoders (2015, background): “This “what”/“where” concept also links to invariant feature hierarchies learning schemes [4, 5, 8, 9, 10, 21, 22].”
End-to-end Optimized Image Compression (2016, methodology): “…synthesis transform is meant to function as an approximate inverse transformation, so we construct it by applying a principle known from the LISTA algorithm (Gregor and LeCun, 2010) to the ﬁxed point iteration previously used to invert the GDN transform (Ball ´ e, Laparra, and Simoncelli, 2015).”
Interpretable Recurrent Neural Networks Using Sequential Sparse Recovery (2016, background): “Gregor and LeCun [1] proposed the learned iterative soft-thresholding algorithm (LISTA) for sparse coding, which uses learned encoders and decoders to increase the speed and performance of the original ISTA algorithm.”
Geometric Deep Learning: Going beyond Euclidean data (2016, methodology): “By unrolling power iterations, one obtains a Graph Neural Network architecture whose parameters can be learnt with backpropagation from labeled examples, similarly to the Learnt Sparse Coding paradigm [91].”
Unsupervised Feature Learning from Temporal Data (2015, background): “Including the rectifying nonlinearity becomes critical for learning sparse inference in a hugely redundant dictionary, e.g. convolutional dictionaries Gregor & LeCun (2010).”
End-to-end optimization of nonlinear transform codes for perceptual quality (2016, methodology): “It is similar in spirit to the LISTA algorithm [6], in that it uses the parametric form of the inversion iteration, but unties its parameters from their original values for faster convergence.”
Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network (2016, methodology): “al [44] trained a cascaded sparse coding network from end to end inspired by LISTA (Learning iterative shrinkage and thresholding algorithm) [16] to fully exploit the natural sparsity of images.”
Tagger: Deep Unsupervised Perceptual Grouping (2016, methodology): “The datasets also contain labels and ground through pixels for digits for segmentation evaluation.”
Neural Expectation Maximization (2017, methodology): “Here the unrolled graph of a ﬁxed number of ISTA iterations is replaced by a recurrent neural network that parametrizes the gradient computations and is trained to predict the sparse codes directly [9].”
Deep Learning of Representations: Looking Forward (2013, methodology): “…to the parameters 7 To define energy, we write probability as the normalized exponential of minus the energy. good news is that for sparse coding, MAP inference is a convex optimization problem for which several fast approximations have been proposed (Mairal et al., 2009; Gregor and LeCun, 2010a).”
Unsupervised Learning of Spatiotemporally Coherent Metrics (2014, background): “Including the rectifying nonlinearity becomes critical for learning sparse inference in a hugely redundant dictionary, e.g. convolutional dictionaries Gregor & LeCun (2010).”
Understanding Deep Architectures using a Recursive Convolutional Network (2013, background): “This novel recursive, convolutional architecture is reminiscent of LISTA [9], but without a direct projection from the input to each hidden layer.”
Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations (2017, methodology): “On the other hand, many deep network designs were inspired by optimization algorithms, such as the network LISTA (Gregor & LeCun, 2010) and the ADMM-Net (Yang et al., 2016).”