High-Performance Neural Networks for Visual Object Classification

By Dan Claudiu Ciresan, Ueli Meier, Jonathan Masci, Luca Maria Gambardella, J. Urgen Schmidhuber. 2011

329 citations recorded by the source corpus.

We present a fast, fully parameterizable GPU implementation of Convolutional Neural Network variants. Our feature extractors are neither carefully designed nor pre-wired, but rather learned in a supervised way. Our deep hierarchical architectures achieve the best published results on benchmarks for object classification (NORB, CIFAR10) and handwritten digit recognition (MNIST), with error rates of 2.53%, 19.51%, 0.35%, respectively. Deep nets trained by simple back-propagation perform better than more shallow ones. Learning is surprisingly rapid. NORB is completely trained within five epochs. Test error rates on MNIST drop to 2.42%, 0.97% and 0.48% after 1, 3 and 17 epochs, respectively.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets (2015, background): “In the recent years, a lot of progress has been made in scaling up algorithms, by either using alternative hardware such as GPUs (Ciresan et al., 2011) or by taking advantage of large clusters (Recht et al., 2011).”
Perceiver: General Perception with Iterative Attention (2021, background): “ConvNets (Fukushima, 1980; LeCun et al., 1998; Cires¸an et al., 2011; Krizhevsky et al., 2012) have been the dominant family of architectures for perceptual tasks for nearly a full decade, thanks to their good performance and scalability.”
Predicting Parameters in Deep Learning (2013, background): “[3] control the number of parameters by removing connections between layers in a convolutional network at random.”
Quantized Convolutional Neural Networks for Mobile Devices (2015, background): “[3] randomly remove connection to reduce network parameters.”
Data-free Parameter Pruning for Deep Neural Networks (2015, methodology): “Our method can be applied on top of most networks with a fully connected layer to give a smaller network.”
Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels (2020, methodology): “Computer Vision Data augmentation via image transformations has been used to improve generalization since the inception of convolutional networks (Becker & Hinton, 1992; Simard et al., 2003; LeCun et al., 1989; Ciresan et al., 2011; Ciregan et al., 2012).”
Deep Learning using Linear Support Vector Machines (2013, background): “These include (but not limited to) speech (Mohamed et al., 2009; Dahl et al., 2010) and vision (Jarrett et al., 2009; Ciresan et al., 2011; Rifai et al., 2011a; Krizhevsky et al., 2012).”
Learning Invariant Representations with Local Transformations (2012, neutral): “9% deep NN (Ciresan et al., 2011) 80 .”
Improvements to Context Based Self-Supervised Learning (2017, background): “mixture of label preserving transformations [37, 5, 4] we are calling the usual bag of tricks (UBT).”
Deep learning for visual understanding: A review (2016, methodology): “In recent years, various fast GPU implementations of CNN variants were presented, most of them utilize max-pooling strategy [6,24].”
Counterexample-Guided Data Augmentation (2018, background): “Traditional data augmentation schemes [van Dyk and Meng, 2001; Simard et al. , 2003; Cires¸an et al. , 2011; Ciregan et al. , 2012; Krizhevsky et al. , 2012] involve geometric transformations which alter the geometry of the image (e.g., rotation, scaling, cropping or ﬂipping); and photometric…”
Matrix capsules with EM routing (2018, background): “2% CNN of Cireşan et al. (2011) with extra input images & deformations 2.”
Selecting Receptive Fields in Deep Networks (2011, methodology): “This is similar to the structure commonly employed by prior work [4, 12], but here we are able to choose receptive fields that span several feature maps in a deliberate way while also exploiting knowledge of the spatial structure.”
Deep Learning using Support Vector Machines (2013, background): “These include (but not limited to) speech (Mohamed et al., 2009; Dahl et al., 2010) and vision (Jarrett et al., 2009; Ciresan et al., 2011; Rifai et al., 2011; Krizhevsky et al., 2012).”
A Spike and Slab Restricted Boltzmann Machine (2011, methodology): “We acknowledge NSERC, FQRNT, RQCHP and Compute Canada for their financial and computational support; Chris Williams for pointing out the connection between the PoPPCA model and the ssRBM; and Charlie Tang for correcting a typo in an earlier manuscript.”