Sparse Feature Learning for Deep Belief Networks

By Marc'Aurelio Ranzato, Y-Lan Boureau, Yann LeCun. 2007

950 citations recorded by the source corpus.

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

Generalized Denoising Auto-Encoders as Generative Models (2013, background): “Some of the first ideas about the probabilistic interpretation of auto-encoders were proposed by Ranzato et al. (2008): they were viewed as approximating an energy function through the reconstruction error, i.”
Representation Learning: A Review and New Perspectives (2012, background): “Sparsity in the representation can be achieved by penalizing the hidden unit biases (making these additive offset parameters more negative) [161], [134], [71], [118] or by directly penalizing the output of the hidden unit activations (making them closer to their saturating value at 0) [162], [123], ”
Compressing Neural Networks with the Hashing Trick (2015, neutral): “, 2011a) or through specialized regularization (Chen et al., 2014a; Boureau et al., 2008).”
Fast Inference in Sparse Coding Algorithms with Applications to Object Recognition (2010, methodology): “r recognitionby comparingto other unsupervised feature extraction algorithms, Principal Components Analysis (PCA), Restricted Boltzman Machine (RBM) [16], and Sparse Encoding Symmetric Machine (SESM) [15]. Then, we compare the recognition accuracy and inference time of PSD feedforwardapproximationto”
A practical tutorial on autoencoders for nonlinear feature fusion: Taxonomy, models, software and guidelines (2018, methodology): “For example, applying a sparsifying logistic activation function in the encoding layer of a similar energy-based model, which forces a low activation average [67], or using a Sparse Encoding Symmetric Machine [68] which optimizes a loss function with a diﬀerent sparsifying penalty.”
Generalizing Pooling Functions in Convolutional Neural Networks: Mixed, Gated, and Tree (2015, neutral): “The specific choices of average pooling [18, 19] and max pooling [28] have been widely used in many CNN-like architectures; [3] includes a theoretical analysis (albeit one based on assumptions that do not hold here).”
Deep Learning of Representations: Looking Forward (2013, background): “It was previously believed (Ranzato et al., 2008), including by the author himself, that reconstruction error should only be small where the estimated density has a peak, e.g., near the data.”
Learning with Hierarchical-Deep Models (2013, background): “Index Terms—Deep networks, deep Boltzmann machines, hierarchical Bayesian models, one-shot learning Ç”
DeepTox: Toxicity Prediction using Deep Learning (2016, background): “Using ReLUs in DNNs leads to sparse input representations , which are robust against noise and advantageous for classifiers because classification is more likely to be easier in higher - dimensional spaces ( Ranzato et al .”
The Utility of Sparse Representations for Control in Reinforcement Learning (2018, background): “…for dictionary learning (Mairal et al. 2009; Mairal et al. 2010; Le, Kumaraswamy, and White 2017) or for general sparse distributions (Ol-shausen and Field 1997; Olshausen 2002; Teh et al. 2003; Ranzato et al. 2006; Ranzato, Boureau, and LeCun 2007; Lee et al. 2008), like Boltzmann machines.”
A tutorial survey of architectures, algorithms, and applications for deep learning (2014, neutral): “[29] developed the sparse encoding symmetric machine (SESM), which has a very similar architecture to RBMs as building blocks of a DBN.”
A Survey on Graph Neural Networks for Time Series: Forecasting, Classification, Imputation, and Anomaly Detection (2023, background): “To achieve this, these frameworks often include constraints and regularization, such as enforcing a low-dimensional embedded code [147] or using variational objectives [148].”