Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognition

By Max Jaderberg, Karen Simonyan, Andrea Vedaldi, Andrew Zisserman. 2014

996 citations recorded by the source corpus.

In this work we present a framework for the recognition of natural scene text. Our framework does not require any human-labelled data, and performs word recognition on the whole image holistically, departing from the character based recognition systems of the past. The deep neural network models at the centre of this framework are trained solely on data produced by a synthetic text generation engine -- synthetic data that is highly realistic and sufficient to replace real data, giving us infinite amounts of training data. This excess of data exposes new possibilities for word recognition models, and here we consider three models, each one "reading" words in a different way: via 90k-way dictionary encoding, character sequence encoding, and bag-of-N-grams encoding. In the scenarios of language based and completely unconstrained text recognition we greatly improve upon state-of-the-art performance on standard datasets, using our fast, simple machinery and requiring zero data-acquisition costs.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Spatial Transformer Networks (2015, methodology): “This is the character sequence model used in [16], where each classifier includes a null-character output to model variable length sequences.”
Deep Structured Output Learning for Unconstrained Text Recognition (2014, methodology): “We now review the component CNN models, originally presented in our tech report Jaderberg et al. (2014a), that form the basis of our joint model in Section 4.”
Speeding up Convolutional Neural Networks with Low Rank Expansions (2014, background): “The combination of an endto-end learning system with minimal need for human design decisions, and the ability to efficiently train large and complex models, have allowed them to achieve state-of-the-art performance in a number of benchmarks [10, 14, 19, 33, 37, 38].”
GIT: A Generative Image-to-text Transformer for Vision and Language (2022, background): “The other is to fine-tune the model on two large scene text datasets: MJSynth (MJ) (Jaderberg et al., 2014; 2016) and SynthText (ST) (Gupta et al.”
AutoML: A Survey of the State-of-the-Art (2019, background): “Moreover, natural scene text can also be generated in [51], [52].”
A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective (2018, background): “Synthetic Data Generative Adversarial Networks [57], [58], [59], [60], [61], [62] Policies [63], [64] Image [65], [66], [67], [68], [69], [70], [71] Text [72], [73], [74]”
Synthetic Data for Text Localisation in Natural Images (2016, methodology): “Text spotting is limited by the detection stage, as stateof-the-art cropped word image recognition accuracy is over 98% [19].”
Compression of fully-connected layer in neural network by Kronecker product (2015, methodology): “…Neural Network by Kronecker Product Shuchang Zhou State Key Laboratory of Computer Architecture Institute of Computing Technology Chinese Academy of Sciences, Beijing, China and Megvii Inc. shuchang.zhou@gmail.com Jia-Nan Wu Department of Computer Science & Engineering Shanghai Jiao Tong University”
Smart Augmentation Learning an Optimal Data Augmentation Strategy (2017, methodology): “Unlike the approach of image blending in [10], we do not arbitrarily or randomly blend images.”
Reading Text in the Wild with Convolutional Neural Networks (2014, methodology): “The lack of full word image samples has caused previous work to rely on character classifiers instead (as character data is plentiful), or this deficit in training data has been mitigated bymining for data or having access to large proprietary datasets (Bissacco et al. 2013; Goodfellow et al. 2013; ”
Use What You Have: Video retrieval using representations from collaborative experts (2019, methodology): “Optical Character Recognition is done in two stages: (1) Each frame is passed through the Pixel Link [10] text detection model to extract bounding boxes for text; (2) The image region of each box is passed through a model [32] that has been trained for scene text recognition on the Synth90K dataset[”
COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Images (2016, methodology): “Other scene text datasets include IIIT 5k-word [13] which contains 3000 cropped word images of scene text downloaded from Google image search as well as Synth90k [5], a dataset of 9 million cropped word images that have been synthetically generated to train systems for character recognition.”
Flowing ConvNets for Human Pose Estimation in Videos (2015, background): “The power of ConvNets has been demonstrated in a wide variety of vision tasks – object classification and detection [11, 21, 28, 40], face recognition [32], text recognition [15, 16], video action recognition [20, 30] and many more [7, 22, 25].”
Inference Compilation and Universal Probabilistic Programming (2016, methodology): “This corresponds to the well-known technique of adding synthetic variations to training data for transformation invariance, as used by Simard et al. (2003), Varga and Bunke (2003), Jaderberg et al. (2014), and many others.”
Exploring Invariances in Deep Convolutional Neural Networks Using Synthetic Images (2014, background): “Finally, several approaches had used synthetic training data for tasks other than object detection, for example, Jaderberg et al. (2014) recently proposed a synthetic text generation engine to perform text recognition in natural scenes.”
Mask TextSpotter: An End-to-End Trainable Neural Network for Spotting Text with Arbitrary Shapes (2018, methodology): “They first use a scene text detector [21, 30, 29] to localize text instances and then use a text recognizer [20, 44] to obtain the recognized text.”
Focusing Attention: Towards Accurate Text Recognition in Natural Images (2017, background): “[11] and 4-million synthetic pixel-wise labeled word instances (excluding the images that contain non-alphanumeric characters) cropped from 80-thousand images released by Gupta et al .”
ASTER: An Attentional Scene Text Recognizer with Flexible Rectification (2019, methodology): “Some methods use convolutional neural networks [33], [34] (CNNs) to localize and recognize characters [29], [62], [70].”