End-to-end scene text recognition

By Kai Wang, Boris Babenko, Serge J. Belongie. 2011

1,305 citations recorded by the source corpus.

This paper focuses on the problem of word detection and recognition in natural images. The problem is significantly more challenging than reading text in scanned documents, and has only recently gained attention from the computer vision community. Sub-components of the problem, such as text detection and cropped image word recognition, have been studied in isolation [7, 4, 20]. However, what is unclear is how these recent approaches contribute to solving the end-to-end problem of word recognition. We fill this gap by constructing and evaluating two systems. The first, representing the de facto state-of-the-art, is a two stage pipeline consisting of text detection followed by a leading OCR engine. The second is a system rooted in generic object recognition, an extension of our previous work in [20]. We show that the latter approach achieves superior performance. While scene text recognition has generally been treated with highly domain-specific methods, our results demonstrate the suitability of applying generic computer vision methods. Adopting this approach opens the door for real world scene text recognition to benefit from the rapid advances that have been taking place in object recognition.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Deep Structured Output Learning for Unconstrained Text Recognition (2014, methodology): “We follow the standard evaluation protocol deﬁned by Wang et al. (2011) and perform recognition on the words containing only alphanumeric characters and at least three characters.”
Speeding up Convolutional Neural Networks with Low Rank Expansions (2014, background): “test set is the collection of 5379 cropped characters from the ICDAR 2003 training set after removing all non-alphanumeric characters as in [3, 41].”
GIT: A Generative Image-to-text Transformer for Vision and Language (2022, background): “, 2012), Street View Text (SVT) (Wang et al., 2011), Street View Text-Perspective (SVTP) (Phan et al.”
Learning Deep Features for Discriminative Localization (2015, neutral): “Furthermore we demonstrate that the CAM localization technique generalizes to other visual recognition tasks i.e., our technique produces generic localizable deep features that can aid other researchers in understanding the basis of discrimination used by CNNs for their tasks.”
Synthetic Data for Text Localisation in Natural Images (2016, background): “[42], where all words that are either less than three characters long or contain non-alphanumeric characters are ignored.”
Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognition (2014, methodology): “there are some publicly available datasets from ICDAR [13, 15, 16, 25], the Street View Text (SVT) dataset [26] and others, the number of full word image samples is only in the thousands, and the vocabulary is very limited.”
Reading Text in the Wild with Convolutional Neural Networks (2014, methodology): “2011), the Street View Text (SVT) dataset (Wang et al. 2011), the IIIT5k dataset (Mishra et al.”
End-to-End Text Recognition with Hybrid HMM Maxout Models (2013, methodology): “However, despite the argument for the validity of task-speciﬁc lexicon use in (Wang et al., 2011), it is clear that we ultimately wish to recognize text with a very general lexicon.”
Scene Text Detection and Recognition: The Deep Learning Era (2018, methodology): “In Wang et al. [142], characters are considered as a special case in object detection and detected by a nearest neighbor clas-siﬁer trained on HOG features [19] and then grouped into words through a Pictorial Structure (PS) based model [26].”
Towards End-to-End Text Spotting with Convolutional Recurrent Neural Networks (2017, methodology): “Datasets The commonly used datasets for scene text spotting include ICDAR2015 [14], ICDAR2011 [22] and Street View Text (SVT) [28].”
Recognizing Text with Perspective Distortion in Natural Scenes (2013, methodology): “It has been shown that scene characters can be extracted as MSERs [11, 12].”
ASTER: An Attentional Scene Text Recognizer with Flexible Rectification (2019, methodology): “Following [60], we discard words that contain non-alphanumeric characters or have less than three characters.”
A Unified Framework for Multioriented Text Detection and Recognition (2014, methodology): “As stated in [22], in conventional end-to-end scene text recognition systems [10], [21], decisions of different components in the pipeline are sequential and isolated, which makes it impossible to exploit feedback information between different components.”
Improving patch-based scene text script identification with ensembles of conjoined networks (2016, methodology): “The evaluation protocol is similar to the one used in other end-to-end scene text recognition datasets [66, 61].”
End-to-end text recognition with convolutional neural networks (2012, methodology): “Benchmark I-5 I-20 I-50 I-FULL SVT Our approach .76 .74 .72 .67 .46 Wang, et al. [18] .72 .70 .68 .51 .38 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 82 84 86 88 90 92 94 96 98 Relative Model Size C la ss i! ca ti o n A cc u ra cy 62−way accuracy of recognition module 2−way accuracy of detection module C180 C”
ICDAR 2013 Robust Reading Competition (2013, methodology): “[19]) • Weakly Contextualised: a vocabulary of all words in the training/test set • Generic: a generic vocabulary of about 90K words derived from the dataset4 of Jaderberg et al.”