Supporting Online Material for Reducing the Dimensionality of Data with Neural Networks

By Geoffrey E. Hinton, Ruslan Salakhutdinov. 2006

Subfields: Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Probabilistic & Bayesian Methods

Citations in the corpus

Adam: A Method for Stochastic Optimization (2014, methodology): “SGD proved itself as an efﬁcient and effective optimization method that was central in many machine learning success stories, such as recent advances in deep learning (Deng et al., 2013; Krizhevsky et al., 2012; Hinton & Salakhutdinov, 2006; Hinton et al., 2012a; Graves et al., 2013).”
Object Detection via a Multi-region and Semantic Segmentation-Aware CNN Model (2015, neutral): “The last two years, huge improvements have been observed on this task thanks to the recent advances of deep learning community [18, 1, 14].”
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks (2013, methodology): “Next, we apply these solutions to investigate the commonly used greedy layer-wise pretraining strategy for training deep networks [17, 18], and recover conditions under which such pretraining speeds learning.”
Improving neural networks by preventing co-adaptation of feature detectors (2012, methodology): “Performance on the test set can be greatly improved by enhancing the training data with transformed images [5] or by wiring knowledge about spatial transformations into a convolutional neural network [17] or by using generative pre-training to extract useful features from the training images without”
Growing a Brain: Fine-Tuning by Increasing Model Capacity (2017, methodology): “This approach was pioneered in [14] by transferring C1-C2-C3-C4-C5 FC6 FC7 FC8 Source task”
Contrastive Multiview Coding (2019, methodology): “such as object recognition. Representations better suited to such tasks have been learnt using deep neural networks, starting with seminal techniques such as Boltzmann machines [55, 50], autoencoders [23], variational autoencoders [30], generative adversarial networks [20] and autoregressive models ”
Neural Discrete Representation Learning (2017, background): “Learning representations with continuous features have been the focus of many previous work [16, 39, 6, 9] however we concentrate on discrete representations [27, 33, 8, 28] which are potentially a more natural ﬁt for many of the modalities we are interested in. Language is inherently discrete,…”
Learning Features by Watching Objects Move (2016, methodology): “ing the input [4, 20, 44], predicting the pixels of the next frame in a video stream [17], metric learning on object track endpoints [46], temporally ordering shuffled frames from a video [29], and spatially ordering patches from a static image [8, 30].”
Context Encoders: Feature Learning by Inpainting (2016, background): “The context encoder is closely related to autoencoders [3, 20], as it shares a similar encoder-decoder architecture.”
Learning to generate chairs with convolutional neural networks (2014, background): “Prominent examples from this line of work are restricted Boltzmann machines (RBMs) [13] and Deep Boltz-mann Machines (DBMs) [26], as well as the plethora of models derived from them [12, 22, 20, 29, 23].”
Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling (2020, background): “Energy-based models [25–33] have gained signiﬁcant attention in recent years.”
Image-to-Image Translation with Conditional Adversarial Networks (2016, background): “Many previous solutions [40, 52, 27, 60, 56] to problems in this area have used an encoder-decoder network [24].”
Generating Images with Perceptual Similarity Metrics based on Deep Networks (2016, methodology): “Autoencoders [6] have been widely used for unsupervised learning and generative modeling, too.”
Generative Visual Manipulation on the Natural Image Manifold (2016, background): “, [19]) and their deep variants [20], auto-encoders [19,21] and more recently, stochastic neural networks [22,3,23] and deterministic networks [24].”
Pedestrian Detection with Unsupervised Multi-stage Feature Learning (2012, methodology): “In [16], it was shown that unsupervised learning can be used to train deep hierarchical models and the final representation achieved is actually useful for a variety of different tasks [32, 24, 4].”
From Pixels to Torques: Policy Learning with Deep Dynamical Models (2015, methodology): “Similar pre-training routines are found in Hinton & Salakhutdinov (2006) where a restricted Boltzmann machine is used instead of PCA.”
Local Aggregation for Unsupervised Learning of Visual Embeddings (2019, background): “Unsupervised learning methods span a very broad spectrum of approaches going back to the roots of artificial neural networks [57, 43, 3, 62, 30, 45, 31, 33, 27], and are too numerous to fully review here.”