Reading Text in the Wild with Convolutional Neural Networks

By Max Jaderberg, Karen Simonyan, Andrea Vedaldi, Andrew Zisserman. 2014

1,217 citations recorded by the source corpus.

In this work we present an end-to-end system for text spotting—localising and recognising text in natural scene images—and text based image retrieval. This system is based on a region proposal mechanism for detection and deep convolutional neural networks for recognition. Our pipeline uses a novel combination of complementary proposal generation techniques to ensure high recall, and a fast subsequent filtering stage for improving precision. For the recognition and ranking of proposals, we train very large convolutional neural networks to perform word recognition on the whole proposal region at the same time, departing from the character classifier based systems of the past. These networks are trained solely on data produced by a synthetic text generation engine, requiring no human labelled data. Analysing the stages of our pipeline, we show state-of-the-art performance throughout. We perform rigorous experiments across a number of standard end-to-end text spotting benchmarks and text-based image retrieval datasets, showing a large improvement over all previous methods. Finally, we demonstrate a real-world application of our text spotting system to allow thousands of hours of news footage to be instantly searchable via a text query.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Deep Structured Output Learning for Unconstrained Text Recognition (2014, methodology): “Previous methods using CNNs for word recognition (discussed in more detail in section Section 2) has either constrained (Jaderberg et al. (2014b)) or heavily weighted (Bissacco et al. (2013)) the recognition results to be from a dictionary of known words.”
Learning from Simulated and Unsupervised Images through Adversarial Training (2016, background): “Many efforts have explored using synthetic data for various prediction tasks, including gaze estimation [40], text detection and classification in RGB images [8, 14], font recognition [39], object detection [9, 24], hand pose estimation in depth images [35, 34], scene recognition in RGB-D [10], sema”
GIT: A Generative Image-to-text Transformer for Vision and Language (2022, methodology): “MJ and ST indicate the MJSynth (MJ) [45, 46] and SynthText (ST) [33] datasets used for training scene text recognition models.”
Recent advances in convolutional neural networks (2015, methodology): “In [165], the major subtasks of its proposed system, namely text bounding box filtering, text bounding box regression, and text recognition are each tackled”
Synthetic Data for Text Localisation in Natural Images (2016, methodology): “[20], it replaces the region proposal stage which typically takes about 3 seconds per image.”
COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Images (2016, methodology): “The problem of detecting and recognizing text in scenes has received increasing attention from the computer vision community in recent years [1, 6, 14, 19].”
Scene Text Visual Question Answering (2019, methodology): “Text recognition methods such as the one presented in [22] propose recognizing text at the word level as a classification problem (word spotting) from a 90K English words vocabulary.”
Reading Scene Text in Deep Convolutional Sequences (2015, methodology): “This problem includes two sub tasks, namely text detection [16, 34, 35, 15, 24] and text-line/word recognition[18, 1, 17, 4, 33].”
Decoupled Attention Network for Text Recognition (2019, background): “(Jaderberg et al. 2016) regraded scene text recognition as a 90k-class classiﬁcation task.”
Arbitrary-Oriented Scene Text Detection via Rotation Proposals (2017, methodology): “[41] extends the region proposal method and applies the Edge Boxes method [39] to perform text detection.”
A Multi-Object Rectified Attention Network for Scene Text Recognition (2019, methodology): “More-over, methods based on convolutional neural networks [3, 22, 50] have been broadly applied.”
Towards End-to-End Text Spotting with Convolutional Recurrent Neural Networks (2017, methodology): “[11] use an ensemble model to generate text proposals, and then adopt the word classifier in [10] for recognition.”
TextSnake: A Flexible Representation for Detecting Text of Arbitrary Shapes (2018, background): “In the past few years, the most prominent trend in the area of scene text detection is the transfer from conventional methods [3, 24] to deep learning based methods [13, 12, 17, 47, 29].”
Real-Time Scene Text Detection With Differentiable Binarization and Adaptive Scale Fusion (2022, background): “[21] first generated word candidates with a proposal generator.”
TextBoxes++: A Single-Shot Oriented Scene Text Detector (2018, methodology): “Other popular examples of this type are the works in [19]–[23]; 2) Word-based: Words are directly extracted in the similar manner as general object detection [8], [24]–[26].”
Detecting Text in Natural Image with Connectionist Text Proposal Network (2016, background): “Reading text in natural image has recently attracted increasing attention in computer vision [8,14,15,10,35,11,9,1,28,32].”