An empirical evaluation of deep architectures on problems with many factors of variation

By Hugo Larochelle, Dumitru Erhan, Aaron C. Courville, James Bergstra, Yoshua Bengio. 2007

1,201 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

FitNets: Hints for Thin Deep Nets (2014, background): “Nevertheless, training deep architectures has proven to be challenging (Larochelle et al., 2007; Erhan et al., 2009), since they are composed of successive non-linearities and, thus result in highly non-convex and non-linear functions.”
Pylearn2: a machine learning research library (2013, methodology): “These include the datasets used for DARPA’s unsupervised and transfer learning challenge [25] , the dataset used for the NIPS 2011 workshops challenge in representation learning [32], the C IFAR-10 and CIFAR-100 datasets [29], the MNIST dataset [34], some of the MNIST variations dataset s [31], the ”
Wide Residual Networks (2016, neutral): “Our code is available at https://github.com/szagoruyko/wide-residual-networks.”
On Network Design Spaces for Visual Recognition (2019, background): “A possible approach for comparing networks from two different model families is to first tune their hyperparameters [16].”
AutoML: A Survey of the State-of-the-Art (2019, methodology): “At the beginning of the study on hyperparameters search, grid search [93]–[95] is one of the most popular methods as it is simple to implement in parallel and tends to find as good as or better hyperparameters than manual search in the”
Paraphrasing Complex Network: Network Compression via Factor Transfer (2018, methodology): “In (Larochelle et al., 2007), a stacked autoencoder on the MNIST dataset achieved great results with a greedy layerwise approach.”
A Hybrid Differential Evolution Approach to Designing Deep Convolutional Neural Networks for Image Classification (2018, neutral): “Therefore, they are picked as the peer competitors of the proposed algorithm, which are CAE-2 [13], TIRBM [15], PGBM+DN1 [5], ScatNet-2 [2], RandNet-2 [3], PCANet-2 (softmax) [3], LDANet-2 [3], SVM+RBF [8], SVM+Poly [8], NNet [8], SAA-3 [8] and DBN-3 [8].”
Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization (2016, methodology): “For CIFAR-10 and MRBI, R is set to 300 (or 30k total iterations).”
Efficient Benchmarking of Hyperparameter Optimizers via Surrogates (2015, methodology): “(2011)), each being used to classify the mrbi and the convex datasets (Larochelle et al. 2007).”
Group Equivariant Stand-Alone Self-Attention For Vision (2020, methodology): “The rotated MNIST dataset (Larochelle et al., 2007) is a classiﬁcation dataset often used as a standard benchmark for rotation equivariance.”
Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data (2020, methodology): “• We evaluate LieConv on the image classification benchmark dataset rotMNIST (Larochelle et al., 2007), and the regression benchmark dataset QM9 (Blum and Reymond, 2009; Rupp et al.”
Group Equivariant Convolutional Networks (2016, methodology): “The dataset is split into 40k Model Test Error (%) Larochelle et al. (2007) 10.38 ± 0.27 Sohn & Lee (2012) 4.2 Schmidt & Roth (2012) 3.”
Routing Networks with Co-training for Continual Learning (2020, methodology): “We test our proposed method with routing networks and co-training on two standard continual learning benchmarks; MNIST-Rotations (Larochelle et al., 2007) and MNISTPermutations (Kirkpatrick et al.”
Compressing Convolutional Neural Networks in the Frequency Domain (2015, methodology): “All three networks are trained on MNIST , and both FreshNets and HashedNets have 1 / 16 compression on the ﬁrst convolutional layer.”
Hyperparameter Selection for Imitation Learning (2021, methodology): “Previous work has looked into improving the selection of the best configurations using grid search (LeCun et al., 2012), random search (Larochelle et al., 2007; Bergstra & Bengio, 2012) or population based strategies (Jaderberg et al., 2017).”
Harmonic Networks: Deep Translation and Rotation Equivariance (2016, methodology): “We investigate classification on the rotated MNIST dataset (new version) [18] as a baseline.”
A Strategy for an Uncompromising Incremental Learner (2017, methodology): “same label space as D b . In the second case, D ihas new classes that are not seen in D b. Case 1: In this experiment, our base dataset D b is the MNIST-rotated dataset developed by Larochelle et al. [14]. This is used to learn G band N b. This is a dataset that is the same as the MNIST dataset, but”