Sparse deep belief net model for visual area V2

By Honglak Lee, Chaitanya Ekanadham, Andrew Y. Ng. 2007

1,107 citations recorded by the source corpus.

Subfields: Generative Models, Neuroscience

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Generative Models

In Neuroscience

Citations in the corpus

Unsupervised Speech Representation Learning Using WaveNet Autoencoders (2019, background): “have been shown to create a hierarchy of visual atoms [11] that match some properties of the visual cortex [22].”
Representation Learning: A Review and New Perspectives (2012, background): “One of the challenges of representation learning that distinguishes it from other machine learning tasks such as classification is the difficulty in establishing a clear objective, or target for training.”
Building high-level features using large scale unsupervised learning (2011, methodology): “Successful feature learning algorithms and their applications can be found in recent literature using a variety of approaches such as RBMs (Hinton et al., 2006), autoencoders (Hinton & Salakhutdinov, 2006; Bengio et al., 2007), sparse coding (Lee et al., 2007) and K-means (Coates et al., 2011).”
Multimodal Deep Learning (2011, methodology): “We use an extension of RBMs with sparsity [8], which have been shown to be able to learn meaningful features for digits and natural images.”
Video (language) modeling: a baseline for generative models of natural videos (2014, methodology): “Other models attempt to overcome this problem by using priors over the features, such as sparsity (Olshausen & Field, 1997; Kavukcuoglu et al., 2008; Lee et al., 2007).”
A practical tutorial on autoencoders for nonlinear feature fusion: Taxonomy, models, software and guidelines (2018, methodology): “Thecommon way to introduce sparsity in an AE is to add a penalty to the loss function, as proposed in [64] for Deep Belief Networks.”
Deep Multimodal Feature Analysis for Action Recognition in RGB+D Videos (2016, methodology): “It has been shown in [29, 50] that applying sparsity on the features of Y and Z will help to improve the learning capability, especially when components are overcomplete.”
Fast ConvNets Using Group-Wise Brain Damage (2015, background): “The combination of sparsity and deep learning has been investigated within several unsupervised approaches such as sparse autoencoders [5, 7] and sparse deep belief networks [33].”
Deep spatial autoencoders for visuomotor learning (2015, background): “Since our learned feature points correspond to object coordinates, this architecture also addresses the second challenge, since realvalued quantities such as positions are more amenable to control than the types of discrete or sparse features that are more commonly learned with deep learning methods”
Practical Recommendations for Gradient-Based Training of Deep Architectures (2012, methodology): “Several recipes have been proposed to extend this idea to visualize the preferred input of hidden units in layers that follow the ﬁrst one (Lee et al. , 2008; Erhan et al. , 2010a).”
Learning Invariant Representations with Local Transformations (2012, methodology): “By following Lee et al.’s (2008) approach, we can extend our model to sparse TIRBM by adding the following regularizer for a given set of data { v (1) , · · · , v ( N ) } to the negative log-likelihood: where D is a distance function; p is the target sparsity. trix T s also induces a linear…”
Deep Learning of Representations: Looking Forward (2013, methodology): “…of research was kick-started in 2006 by a few research groups, starting with Geoff Hinton’s group, who initially focused on stacking unsupervised representation learning algorithms to obtain deeper representations (Hinton et al., 2006; Bengio et al., 2007; Ranzato et al., 2007; Lee et al., 2008).”
The Utility of Sparse Representations for Control in Reinforcement Learning (2018, background): “…for dictionary learning (Mairal et al. 2009; Mairal et al. 2010; Le, Kumaraswamy, and White 2017) or for general sparse distributions (Ol-shausen and Field 1997; Olshausen 2002; Teh et al. 2003; Ranzato et al. 2006; Ranzato, Boureau, and LeCun 2007; Lee et al. 2008), like Boltzmann machines.”
Beyond Shared Hierarchies: Deep Multitask Learning through Soft Layer Ordering (2017, background): “This is not surprising, as deep vision systems are known to induce a common feature hierarchy, especially within the first couple of layers (Lee et al., 2008; Lecun et al., 2015).”
Learning Intermediate-Level Representations of Form and Motion from Natural Movies (2012, background): “These representations provide a rich description of dynamic natural images and testable hypotheses regarding intermediate-level representation in visual cortex.”