Multi-column deep neural networks for image classification

By Dan Claudiu Ciresan, Ueli Meier, J. Urgen Schmidhuber. 2012

4,054 citations recorded by the source corpus.

Traditional methods of computer vision and machine learning cannot match human performance on tasks such as the recognition of handwritten digits or traffic signs. Our biologically plausible, wide and deep artificial neural network architectures can. Small (often minimal) receptive fields of convolutional winner-take-all neurons yield large network depth, resulting in roughly as many sparsely connected neural layers as found in mammals between retina and visual cortex. Only winner neurons are trained. Several deep neural columns become experts on inputs preprocessed in different ways; their predictions are averaged. Graphics cards allow for fast training. On the very competitive MNIST handwriting benchmark, our method is the first to achieve near-human performance. On a traffic sign recognition benchmark it outperforms humans by a factor of two. We also improve the state-of-the-art on a plethora of common image classification benchmarks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Factorization tricks for LSTM networks (2017, background): “, 2017) or multi-tower (Ciregan et al., 2012) models, the key differences are: (1) input to different groups is different and assumed independent, and (2) instead of computing ensemble output, it is concatenated into independent pieces.”
Sequence to Sequence Learning with Neural Networks (2014, background): “Deep Neural Networks (DNNs) are extremely powerful machine learning models that achieve excellent performance on difficult problems such as speech recognition [13, 7] and visual object recognition [19, 6, 21, 20].”
Training Very Deep Networks (2015, background): “For instance, within just a few years, the top-5 image classiﬁcation accuracy on the 1000-class ImageNet dataset has increased from ∼ 84% [1] to ∼ 95% [2, 3] using deeper networks with rather small receptive ﬁelds [4, 5].”
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification (2015, background): “Convolutional neural networks (CNNs) [19, 18] have demonstrated recognition accuracy better than or comparable to humans in several visual recognition tasks, including recognizing traffic signs [3], faces [34, 32], and handwritten digits [3, 36].”
Convolutional neural networks at constrained time cost (2014, methodology): “In previous usages of max pooling [14, 2], this layer has been playing two different roles: (i) lateral suppression (max filtering) that increases the invariance to small local translation, and (ii) reducing the spatial size of feature maps by subsampling.”
Deeply-Supervised Nets (2014, methodology): “Finally, we note that our approach is independent of techniques such as averaging [22], dropconnect [17], and Maxout [10]; thus, we expect that combining DSN with these techniques might result in even greater classification error reduction.”
Very Deep Convolutional Networks for Large-Scale Image Recognition (2014, methodology): “…networks (ConvNets) have recently enjoyed a great success in large-scale visual recognition [13, 20, 21, 25] which has become possible due to the large public image repositories, such as ImageNet [4], and high-performance computing systems, such as GPUs or large-scale distributed clusters [3].”
OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks (2013, background): “Wealso introduce a novel deep learning approach to localization by learning topredict object boundaries.”
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks (2013, background): “Recently high performance has been demonstrated in deep networks trained from random initial conditions [21, 13, 22, 3, 4, 1, 23], suggesting that deep networks may not be as hard to train as previously thought.”
Adversarial NLI: A New Benchmark for Natural Language Understanding (2019, background): “Whereas it took around 15 years to achieve “near-human performance” on MNIST (LeCun et al., 1998; Cires¸an et al., 2012; Wan et al., 2013) and approximately 7 years to surpass humans on ImageNet (Deng et al., 2009; Russakovsky et al., 2015; He et al., 2016), the GLUE benchmark did not last as long…”
Adaptive Computation Time for Recurrent Neural Networks (2016, positive): “The evidence that increased depth leads to more performant networks is by now inarguable [5, 4, 19, 9], and recent results show that increased sequence length can be similarly beneficial [31, 33, 25].”
What can the brain teach us about building artificial intelligence? (2016, background): “So far, these abilities elude the best deep neural networks for character recognition (Ciresan et al., 2012), which are trained on many examples of each concept and do not flexibly generalize to new tasks.”
Visualizing and Understanding Convolutional Networks (2013, background): “(Ciresan et al., 2012) demonstrate state-of-the-art performance on NORB and CIFAR-10 datasets.”
Randaugment: Practical automated data augmentation with a reduced search space (2019, background): “On MNIST, elastic distortions across scale, position, and orientation have been applied to achieve impressive results [43, 4, 49, 42].”
End-to-End Training of Deep Visuomotor Policies (2015, background): “, 1989; Schmidhuber, 2015), and have recently gained prominence due to excellent results on a number of vision benchmarks (Ciresan et al., 2011; Krizhevsky et al., 2012; Ciresan et al., 2012; Girshick et al., 2014a; Tompson et al., 2014; LeCun et al., 2015; He et al., 2015).”
Some Improvements on Deep Convolutional Neural Network Based Image Classification (2013, background): “[2] D.”
Predicting Parameters in Deep Learning (2013, background): “The use of different preprocessing for different columns of representation is of particular relevance [2].”
Value Iteration Networks (2016, background): “Over the last decade, deep convolutional neural networks (CNNs) have revolutionized supervised learning for tasks such as object recognition, action recognition, and semantic segmentation [4, 13, 7, 17].”