Discriminative Recurrent Sparse Auto-Encoders

By Jason Tyler Rolfe, Yann LeCun. 2013

74 citations recorded by the source corpus.

We present the discriminative recurrent sparse auto-encoder model, comprising a recurrent encoder of rectified linear units, unrolled for a fixed number of iterations, and connected to two linear decoders that reconstruct the input and predict its supervised classification. Training via backpropagation-through-time initially minimizes an unsupervised sparse reconstruction error; the loss function is then augmented with a discriminative term on the supervised classification. The depth implicit in the temporally-unrolled form allows the system to exhibit all the power of deep networks, while substantially reducing the number of trainable parameters. From an initially unstructured network the hidden units differentiate into categorical-units, each of which represents an input prototype with a well-defined class; and part-units representing deformations of these prototypes. The learned organization of the recurrent encoder is hierarchical: part-units are driven directly by the input, whereas the activity of categorical-units builds up over time through interactions with the part-units. Even using a small number of hidden units per layer, discriminative recurrent sparse auto-encoders achieve excellent performance on MNIST.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Interpretable Recurrent Neural Networks Using Sequential Sparse Recovery (2016, positive): “Rolfe and LeCun [2] created networks from ISTA under a nonnegativity constraint on the sparse coefﬁcients.”
Understanding Deep Architectures using a Recursive Convolutional Network (2013, background): “Gregor & LeCun [9] showed how to backpropagate through such a network to give fast approximations to sparse coding known as LISTA.”
On autoencoder scoring (2013, methodology): “…can assign scores, too, if they are trained in a certain way: If noise is added to the input data during training, minimizing squared error is related to performing score matching (Hyvärinen, 2005) in an undirect probabilistic model, as shown by (Swersky et al., 2011) and (Vincent, 2011).”
Global Belief Recursive Neural Networks (2014, background): “[20] unfold the same autoencoder multiple times which gives it more representational power with the same number of parameters.”
Multilingual Deep Learning (2013, background): “[19] proposes a novel variant of auto-encoder called Discriminative Recurrent Sparse Auto-encoders, which allows sharing parameters between successive layers of a deep network.”
AnomalyNet: An Anomaly Detection Network for Video Surveillance (2019, methodology): “tion or called differentiable programming has attracted a lot of interests [25], [42]–[48] thanks to following advantages.”