Supporting Online Material for Reducing the Dimensionality of Data with Neural Networks

By Geoffrey E. Hinton, Ruslan Salakhutdinov. 2006

9,072 citations recorded by the source corpus.

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences (2019, positive): “Orthologous group translation For a given sequence representation function r , we deﬁne a group mean translation T Gr, G → G ( s ) on a sequence s from orthologous group G to group G (cid:48) as follows: This translation subtracts from r ( s ) the averaged sequence representation across all…”
Multi-Level Cause-Effect Systems (2015, positive): “In our experiments, dimensionality reduction with autoencoders (Hinton and Salakhutdinov, 2006) followed by kernel density estimation worked well.”
Pre-training Graph Neural Networks with Kernels (2018, positive): “The breakthrough that allowed to effectively train large-scale “deep” networks has been the introduction of an unsupervised pre-training phase [Hinton and Salakhutdinov, 2006], in which the network is trained to build a generative model of the data, which can be subsequently reﬁned using a…”
Norm-preserving Orthogonal Permutation Linear Unit Activation Functions (OPLU) (2016, neutral): “Early DNNs had 4-6 hidden layers [1], [2].”
A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning (2010, neutral): “…algorithms [Bertsekas and Tsitsiklis, 1996, Sutton and Barto, 1998], learning methods for Boltzmann machines and deep belief networks [Younes, 1989, Hinton and Salakhutdinov, 2006] and parameter estimation for nonlinear state space models [Poyiadjis et al., 2005, Martinez–Cantin et al., 2006].”
DeepMPC: Learning Deep Latent Features for Model Predictive Control (2015, positive): “In the recent past, feature learning methods have proven effective for learning latent task-specific features across many domains [3, 19, 24, 9, 28].”
Variable Rate Image Compression with Recurrent Neural Networks (2015, positive): “More recently, autoencoder architectures (Hinton & Salakhutdinov, 2006) have become viable as a means of implementing end-to-end compression.”
Taskonomy: Disentangling Task Transfer Learning (2018, positive): “Autoencoding [14] is a nonlinear generalization of PCA that was originally proposed with transfer learning in mind: better downstream performance through autoencoding pretraining.”
Neurogenesis deep learning: Extending deep networks to accommodate new classes (2016, positive): “However, we did not otherwise normalize the characters within classes, so the variation in scale and location within the 28 x 28 frame is much greater than the MNIST data.”
Disentangling the independently controllable factors of variation by interacting with the world (2018, positive): “The canonical deep learning framework to learn representations is the autoencoder framework [Hinton and Salakhutdinov, 2006].”
Deep Learning of Representations: Looking Forward (2013, neutral): “From a computation point of view, how do we scale the recent successes of deep learning to much larger models and huge datasets, such that the models are actually richer and capture a very large amount of information?”
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (2015, neutral): “Today’s faster computers, such as GPUs, mitigate the Fundamental Deep Learning Problem for FNNs [181, 34, 198, 38, 40].”
Fixed-point feedforward deep neural network design using weights +1, 0, and −1 (2014, positive): “This database has been widely used for evaluation of various image classifiers [2], [15], [16].”
DeepID-Net: multi-stage and deformable deep convolutional neural networks for object detection (2014, neutral): “. 2. Related Work It has been proved that deep models are potentially more capable than shallow models in handling complex tasks [4]. Deep models have achieved spectacular progress in computer vision [26, 27, 43, 28, 30, 37, 29, 63, 33, 50, 18, 42]. Because of its power in learning feature represent”
Pre-trained models for natural language processing: A survey (2020, neutral): “As early as 2006, the breakthrough of deep learning came with greedy layer-wise unsupervised pre-training followed by supervised fine-tuning [61].”
Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice (2017, positive): “Indeed the very genesis of deep learning rested upon the initial observation that unsupervised pre-training provides a good set of initial weights for subsequent fine-tuning through backpropagation [7].”
Divergence Triangle for Joint Training of Generator Model, Energy-Based Model, and Inferential Model (2018, neutral): “Layer-wise training dates back to initializing deep neural networks by Restricted Boltzmann Machines to overcome optimization hurdles [48], [49] and has been resurrected in progressive GANs [47], albeit the order of layer transitions is reversed such that top layers are trained first.”