Deep, Big, Simple Neural Nets for Handwritten Digit Recognition

By Dan Claudiu Ciresan, Ueli Meier, Luca Maria Gambardella, J. Urgen Schmidhuber. 2010

1,017 citations recorded by the source corpus.

Good old online backpropagation for plain multilayer perceptrons yields a very low 0.35 error rate on the MNIST handwritten digits benchmark. All we need to achieve this best result so far are many hidden layers, many neurons per layer, numerous deformed training images to avoid overfitting, and graphics cards to greatly speed up learning.

Subfields: Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Maxout Networks (2013, background): “(It is possible to get better results on MNIST by augmenting the dataset with transformations of the standard set of images (Ciresan et al., 2010) ) A summary of the best methods on the general MNIST dataset is provided in Table 2.”
Improving neural networks by preventing co-adaptation of feature detectors (2012, background): “Performance on the test set can be greatly improved by enhancing the training data with transformed images [5] or by wiring knowledge about spatial transformations into a convolutional neural network [17] or by using generative pre-training to extract useful features from the training images without”
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (2020, background): “Since late 2000s, researchers started to leverage GPUs to accelerate neural nets [70, 57, 71].”
MixMatch: A Holistic Approach to Semi-Supervised Learning (2019, background): “For example, in image classification, it is common to elastically deform or add noise to an input image, which can dramatically change the pixel content of an image without altering its label [7, 43, 10].”
Representation Learning: A Review and New Perspectives (2012, methodology): “[47], a large number of labeled examples, artificially transformed”
Multi-column deep neural networks for image classification (2012, methodology): “Just like for MNIST, the initial learning rate 0.001 decays by a factor of 0.993 after every epoch.”
Building high-level features using large scale unsupervised learning (2011, methodology): “1 Our input data are 200x200 images, much larger than typical 32x32 images used in deep learning and unsupervised feature learning (Krizhevsky, 2009; Ciresan et al., 2010; Le et al., 2010; Coates et al., 2011).”
Deep Learning with Limited Numerical Precision (2015, neutral): “, 2014), or high-end graphics processors (GPUs) (Ciresan et al., 2010; Krizhevsky et al., 2012),”
Fast image scanning with deep max-pooling convolutional neural networks (2013, background): “Simple GPU-based multilayer perceptrons (MLP) establised new state of the art results [5] on the MNIST handwritten digit dataset [4] when made both deep and large (augmenting the training set by artificial samples helped to avoid overfitting).”
Deep learning in neural networks: An overview (2014, methodology): “Dempster, A. P., Laird, N. M., and Rubin, D. B. (1977). Maximulikelihood from incomplete data via the EM algorithm.”
Understanding Data Augmentation for Classification: When to Warp? (2016, methodology): “This approa ch was further extended to create a simple deep neural network that was only trained on warped data [6], where new warped training data was created for each epoch of the backpropagation algorithm.”
Deep Learning: Methods and Applications (2014, methodology): “Some preliminary results have been discussed in this monograph and in [55, 161, 323, 429].”
Machine learning of molecular electronic properties in chemical compound space (2013, methodology): “The idea of encoding known invariances through such data extension has previously been used to improve prediction accuracy on image classification and handwritten digit recognition data sets [61].”
High-Performance Neural Networks for Visual Object Classification (2011, methodology): “These deformations improve recognition rates for digits that are intrinsically 2D (Cire¸san et al. , 2010), but seem inadequate for 3D objects.”
Norm-preserving Orthogonal Permutation Linear Unit Activation Functions (OPLU) (2016, background): “Early DNNs had 4-6 hidden layers [1], [2].”
APAC: Augmented PAttern Classification with Neural Networks (2015, background): “[3, 2], where they claim that the original data can be used for validation.”
Deep Image: Scaling up Image Recognition (2015, background): “…itself: simple, scalable algorithms, given enough data and computational resources, dominate many ﬁelds, including visual object recognition (Ciresan et al., 2010), (Krizhevsky et al., 2012), (Szegedy et al., 2014), speech recognition (Dahl et al., 2012), (Han-nun et al., 2014) and natural…”