Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks

By Ian J. Goodfellow, Yaroslav Bulatov, Julian Ibarz, Sacha Arnoud, Vinay D. Shet. 2013

757 citations recorded by the source corpus.

Abstract: Recognizing arbitrary multi-character text in unconstrained natural photographs is a hard problem. In this paper, we address an equally hard sub-problem in this domain viz. recognizing arbitrary multi-digit numbers from Street View imagery. Traditional approaches to solve this problem typically separate out the localization, segmentation, and recognition steps. In this paper we propose a unified approach that integrates these three steps via the use of a deep convolutional neural network that operates directly on the image pixels. We employ the DistBelief implementation of deep neural networks in order to train large, distributed neural networks on high quality images. We find that the performance of this approach increases with the depth of the convolutional network, with the best performance occurring in the deepest architecture we trained, with eleven hidden layers. We evaluate this approach on the publicly available SVHN dataset and achieve over $96\%$ accuracy in recognizing complete street numbers. We show that on a per-digit recognition task, we improve upon the state-of-the-art, achieving $97.84\%$ accuracy. We also evaluate this approach on an even more challenging dataset generated from Street View imagery containing several tens of millions of street number annotations and achieve over $90\%$ accuracy. To further explore the applicability of the proposed system to broader text recognition tasks, we apply it to synthetic distorted text from reCAPTCHA. reCAPTCHA is one of the most secure reverse turing tests that uses distorted text to distinguish humans from bots. We report a $99.8\%$ accuracy on the hardest category of reCAPTCHA. Our evaluations on both tasks indicate that at specific operating thresholds, the performance of the proposed system is comparable to, and in some cases exceeds, that of human operators.

Subfields: Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Spatial Transformer Networks (2015, methodology): “Size Model 64px 128px Maxout CNN [10] 4.”
Deep Structured Output Learning for Unconstrained Text Recognition (2014, methodology): “(2014a)) for two scenarios: the first was to train a different CNN character classifier for each position in the word being recognised, using the whole image of the word as input to each classifier (an idea also expored by Goodfellow et al. (2013)); the second was to construct a CNN predictor to det”
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems (2016, neutral): “In addition, often in close collaboration with the Google Brain team, more than 50 teams at Google and other Alphabet companies have deployed deep neural networks using DistBelief in a wide variety of products, including Google Search [11], our advertising products, our speech recognition systems [5”
DRAW: A Recurrent Neural Network For Image Generation (2015, methodology): “We used the same preprocessing as (Goodfellow et al., 2013), yielding a 64 × 64 house number image for each training example.”
Speeding up Convolutional Neural Networks with Low Rank Expansions (2014, background): “The combination of an endto-end learning system with minimal need for human design decisions, and the ability to efficiently train large and complex models, have allowed them to achieve state-of-the-art performance in a number of benchmarks [10, 14, 19, 33, 37, 38].”
Deep learning in neural networks: An overview (2014, background): “They also helped to recognise multi-digit numbers in Google Street View images (Goodfellow et al., 2014b), where part of the NN was trained to count visible digits.”
Recent advances in convolutional neural networks (2015, methodology): “2) Text recognition: [159] proposes a CNN model with multiple softmax classifiers in its final layer, which is formulated in such a way that each classifier is responsible for character prediction at each sequential location in the multi-digit input image.”
Dynamic Routing Between Capsules (2017, background): “Hinton et al. propose the task of segmenting and recognizing highly overlapping digits (Hinton et al. [2000] and others have tested their networks in a similar domain (Goodfellow et al. [2013], Ba et al. [2014], Greff et al. [2016]).”
Synthetic Data for Text Localisation in Natural Images (2016, background): “The first is that complex recognition pipelines that recognise text by explicitly combining recognition and detection of individual characters can be replaced by very powerful classifiers that directly map an image patch to words [13, 20].”
Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognition (2014, methodology): “[8] had great success using a CNN with multiple position-sensitive character classifier outputs (closely related to the character sequence model in Sect.”
Predictive Uncertainty Estimation via Prior Networks (2018, methodology): “For both core and additional experiments models were trained on the MNIST [35], SVHN [38] and CIFAR [36] datasets.”
Tagger: Deep Unsupervised Perceptual Grouping (2016, methodology): “Our assumptions are very similar to these approaches, but we allow the model to learn the amortized inference directly (more in line with Goodfellow et al. [7]).”
Motivating the Rules of the Game for Adversarial Example Research (2018, background): “For example, neural networks trained on obfuscated text CAPTCHAs can read obfuscated text better than humans [121].”
Reading Text in the Wild with Convolutional Neural Networks (2014, methodology): “While not performing full scene text recognition, (Goodfellow et al. 2013) had great success using a CNN with multiple position-sensitive character classifier outputs to perform street number recognition.”
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (2015, background): “There are many recent variations and improvements [64, 74, 124, 75, 277, 266, 245].”
Flowing ConvNets for Human Pose Estimation in Videos (2015, background): “The power of ConvNets has been demonstrated in a wide variety of vision tasks – object classification and detection [11, 21, 28, 40], face recognition [32], text recognition [15, 16], video action recognition [20, 30] and many more [7, 22, 25].”
Inference Compilation and Universal Probabilistic Programming (2016, background): “, 2014, 2013), and state-of-the-art results have been obtained by using deep convolutional neural networks (CNNs) (Goodfellow et al., 2013; Stark et al., 2015), at the cost of requiring very large (in the order of millions) labeled training sets for supervised learning.”