Generative models for discovering sparse distributed representations.

By Geoffrey E. Hinton, Zoubin Ghahramani. 1997

Subfields: Generative Models, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Generative Models

In Representation & Self-Supervised Learning

Citations in the corpus

Deep learning in neural networks: An overview (2014, background): “Many UL methods automatically and robustly generate distributed, sparse representations of input patterns (Földiák, 1990; Hinton and Ghahramani, 1997; Lewicki and Olshausen, 1998; Hyvärinen et al., 1999; Hochreiter and Schmidhuber, 1999; Falconbridge et al., 2006) through well-known feature detector”
Lightweight Probabilistic Deep Networks (2018, background): “Later, [12, 22] introduce Gaussian units to represent latent variables that are nonlinearly propagated through network layers.”
Nonlinear Statistical Learning with Truncated Gaussian Graphical Models (2016, methodology): “To overcome the linearity of GGMs, Frey (1997), Hinton & Ghahramani (1997), and Frey & Hinton (1999) proposed to apply nonlinear transformations to Gaussian hidden variables.”
Single Image 3D Interpreter Network (2016, methodology): “ its 2D keypoints, which is more robust to imperfect detection results and can be jointly learned with keypoint estimators. Our work also connects to the traditional eld of vision as inverse graphics [14,21] and analysis by synthesis [58,20,5,52], as we use neural nets to decode latent 3D structure ”
Automatic differentiation in machine learning: a survey (2015, background): “problems is inverse graphics (Horn, 1977; Hinton and Ghahramani, 1997)—or analysis-bysynthesis (Yildirim et al.”
Breaking the Curse of Dimensionality with Convex Neural Networks (2014, methodology): “…as the affine projection of a hypercube; in particular, when using a higher-dimensional distributed representation of points in Rd with elements in [0, 1]r , where r is larger than d (see, e.g., Hinton and Ghahramani, 1997), the underlying polytope that is modelled in Rd happens to be a zonotope.”
Approximate Bayesian Image Interpretation using Generative Probabilistic Graphics Programs (2013, background): “Many pattern recognition and learning techniques can then be used to build classifiers for individual scene elements, and sometimes to learn the features themselves [7, 6].”
Flexible shaping: how learning in small steps helps. (2009, unknown): “In the field of machine learning, this is the standard view of the interaction between unsupervised learning, which represents inputs according to the underlying structure of their statistical distribution, and supervised learning, which uses these representations to perform tasks well (Hinton, Osin”
Bayesian Computation Emerges in Generic Cortical Microcircuits through Spike-Timing-Dependent Plasticity (2013, background): “The concept of a generative model is well known in theoretical neuroscience [26,27], but it has so far primarily been applied in the context of an abstract non-spiking neural circuit architecture.”
Variational Learning in Nonlinear Gaussian Belief Networks (1999, methodology): “The features for the noisy data can be cleaned up if some weight decay is used (Hinton and Ghahramani 1997).”
Unsupervised Learning of Individuals and Categories from Images (2008, background): “Hinton and Ghahramani (1997) also cast sparse representation in a generative modeling framework, but as with Olshausen and Field, they work directly at the image level.”
Bayesian Spiking Neurons I: Inference (2008, background): “This model is called generative because it defines the way that observations (the sensory input) are assumed to be generated (or caused) by the state of the hidden variable (Hinton & Ghahramani, 1997).”
Bilinear Sparse Coding for Invariant Vision (2005, methodology): “Algorithms for redundancy reduction and efficient coding have been the subject of considerable attention (Olshausen & Field, 1996, 1997; Bell & Sejnowski , 1997; Hinton & Ghahramani, 1997; Rao & Ballard, 1999; Lewicki & Sejnowski, 2000; Schwartz & Simoncelli, 2001).”