Reading Scene Text in Deep Convolutional Sequences

By Pan He, Weilin Huang, Yu Qiao, Chen Change Loy, Xiaoou Tang. 2015

321 citations recorded by the source corpus.

We develop a Deep-Text Recurrent Network (DTRN)that regards scene text reading as a sequence labelling problem. We leverage recent advances of deep convolutional neural networks to generate an ordered highlevel sequence from a whole word image, avoiding the difficult character segmentation problem. Then a deep recurrent model, building on long short-term memory (LSTM), is developed to robustly recognize the generated CNN sequences, departing from most existing approaches recognising each character independently. Our model has a number of appealing properties in comparison to existing scene text recognition methods: (i) It can recognise highly ambiguous words by leveraging meaningful context information, allowing it to work reliably without either pre- or post-processing; (ii) the deep CNN feature is robust to various image distortions; (iii) it retains the explicit order information in word image, which is essential to discriminate word strings; (iv) the model does not depend on pre-defined dictionary, and it can process unknown words and arbitrary strings. It achieves impressive results on several benchmarks, advancing the-state-of-the-art substantially.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Chunhua Shen
2. Cong Yao
3. Xiang Bai
4. Jianfei Cai
5. Tong He
6. Minghui Liao
7. Zhi Tian
8. Xin He

In Computer Vision and Pattern Recognition

1. Xiang Bai
2. Chunhua Shen
3. Cong Yao
4. Minghui Liao
5. Yu Qiao
6. Luc Van Gool
7. Dacheng Tao
8. Ming-Hsuan Yang

Citations in the corpus

• Recent advances in convolutional neural networks (2015, methodology): “In [161], CNN extracts rich visual features from character-level image patches obtained via sliding window, and the sequence labelling is carried out by an enhanced RNN variant called Long Short-Term Memory (LSTM) [162].”
• Towards End-to-End Text Spotting with Convolutional Recurrent Neural Networks (2017, unknown): “[7] and Shi et al .”
• A Multi-Object Rectified Attention Network for Scene Text Recognition (2019, background): “Integrating recognition models with recurrent neural networks [17, 41, 42] and attention mechanisms [5, 6, 27, 51] yields better performance for these models.”
• ESIR: End-To-End Scene Text Recognition via Iterative Image Rectification (2018, unknown): “[13], [6] and [36] propose end-to-end systems that use RNNs for visual feature representation and CTC for sequence prediction.”
• Single Shot Text Detector with Regional Attention (2017, methodology): “ modules, and enhances the convolutional features towards text task. age deep features, jointly learned with a classifier, to enable strong representation of text; (ii) sharing convolutional mechanism [19,6] was applied for reducing the computational cost remarkably. With these two improvements, a nu”
• Text-Attentional Convolutional Neural Network for Scene Text Detection (2015, background): “Recently, a number of deep models have been developed for text component filtering/classification [1], [8], [2], or word/character recognition [9], [8], [10], [2], by leveraging advances of deep image representation.”
• Show, Attend and Read: A Simple and Strong Baseline for Irregular Text Recognition (2018, methodology): “Our proposed model is composed of a 31-layer ResNet, an LSTM-based encoder-decoder framework and a tailored 2 -dimensional attention module.”
• Scene Text Detection and Recognition: The Deep Learning Era (2018, methodology): “Now this technique is widely adopted in scene text recognition (Su and Lu, 2014; He et al., 2016; Liu et al., 2016b; Gao et al., 2017; Shi et al., 2017b; Yin et al., 2017).”
• RobustScanner: Dynamically Enhancing Positional Clues for Robust Text Recognition (2020, methodology): “Later, top-down methods [12,40,15,11,24,41,3,42,57,8,7,25,50,35,1,55,45,16] were proposed, in which text sequences are end-to-end predicted without the single character detection.”
• Detecting Text in Natural Image with Connectionist Text Proposal Network (2016, background): “Reading text in natural image has recently attracted increasing attention in computer vision [8, 14, 15, 10, 35, 11, 9, 1, 28, 32].”
• An End-to-End TextSpotter with Explicit Alignment and Attention (2018, methodology): “[9] cast the task of word recognition into a sequence labelling problem, where an encoding-decoding process is introduced by incorporating LSTM [3] and connectionist temporal classification (CTC) [4] into a unified framework.”
• Symmetry-Constrained Rectification Network for Scene Text Recognition (2019, methodology): “One subcategory is based on Connectionist Temporal Classification (CTC) [13, 44, 17] while the other is based on attention decoders [8, 46, 7].”
• ASTER: An Attentional Scene Text Recognizer with Flexible Rectification (2019, methodology): “Their effort was followed by [21], [54], where CNN and RNN are integrated for text recognition.”
• Mask TextSpotter: An End-to-End Trainable Neural Network for Spotting Text with Arbitrary Shapes (2018, methodology): “al. propose a word-based method which treats text recognition as a common English words (90k) classification problem. Sequenceto-sequence methods solve text recognition as a sequence labeling problem. [25], [65], [68] use CNN and RNN to model image features and output the recognized sequences with CT”