Robust Scene Text Recognition with Automatic Rectification

By Baoguang Shi, Xinggang Wang, Pengyuan Lyu, Cong Yao, Xiang Bai. Computer Vision and Pattern Recognition, 2016

638 citations recorded by the source corpus.

Recognizing text in natural images is a challenging task with many unsolved problems. Different from those in documents, words in natural images often possess irregular shapes, which are caused by perspective distortion, curved character placement, etc. We propose RARE (Robust text recognizer with Automatic REctification), a recognition model that is robust to irregular text. RARE is a speciallydesigned deep neural network, which consists of a Spatial Transformer Network (STN) and a Sequence Recognition Network (SRN). In testing, an image is firstly rectified via a predicted Thin-Plate-Spline (TPS) transformation, into a more "readable" image for the following SRN, which recognizes text through a sequence recognition approach. We show that the model is able to recognize several types of irregular text, including perspective text and curved text. RARE is end-to-end trainable, requiring only images and associated text labels, making it convenient to train and deploy the model in practical systems. State-of-the-art or highly-competitive performance achieved on several benchmarks well demonstrates the effectiveness of the proposed model.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/edd0f6d316d33c61a4a69c9262d1b1e07a93bae6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Xiang Bai
2. Chunhua Shen
3. Cong Yao
4. Luc Van Gool
5. Dacheng Tao
6. Yu Qiao
7. Xiaogang Wang
8. Andrew Zisserman

In Media Technology

1. Cong Yao
2. Xiang Bai
3. Minghui Liao
4. Xin He
5. Shangbang Long
6. Lianwen Jin
7. Yuliang Liu
8. Pengyuan Lyu

Citations in the corpus

• Scene Text Visual Question Answering (2019, methodology): “An attention based sequence-to-sequence model is used by [45] and a connectionist temporal clas-sification (CTC) is proposed by [12].”
• Show, Attend and Read: A Simple and Strong Baseline for Irregular Text Recognition (2018, methodology): “Existing irregular text recognizers can be roughly categorized into three groups: rectification based (Shi et al. 2016; 2018; Liu et al. 2016; Liu, Chen, and Wong 2018), attention based (Yang et al. 2017; Cheng et al. 2017) and multi-direction encoding based (Cheng et al. 2018) approaches.”
• ESIR: End-To-End Scene Text Recognition via Iterative Image Rectification (2018, methodology): “5 shows, the proposed network produces clearly better rectifications as compared with rectifications by RARE and ASTER.”
• Recognizing Text-Based Traffic Guide Panels with Cascaded Localization Network (2016, methodology): “To avoid the above numerous local computation, several methods based on recurrent neural network (RNN) with long short-term memory (LSTM) are recently proposed [16,17], which model a word image as an unsegmented sequence and does not require character-level segmentation and recognition.”
• ASTER: An Attentional Scene Text Recognizer with Flexible Rectification (2019, methodology): “This work further extends [55] with the bidirectional decoder.”
• Mask TextSpotter: An End-to-End Trainable Neural Network for Spotting Text with Arbitrary Shapes (2018, methodology): “Besides, different from the previous sequence-based recognition methods [45, 44, 26] which are designed for 1-D sequence, we recognize text via semantic”
• Learning RoI Transformer for Oriented Object Detection in Aerial Images (2019, background): “The STN and deformable modules are widely used for recognition in the field of scene text and aerial images [40, 33, 19, 34, 39].”
• Scene Text Detection and Recognition: The Deep Learning Era (2018, methodology): “(1) CNN+RNN [123] (2) Conv Sequence Learning [28] (3) Sliding Window [158] (1) Attention-based decoding [70](2) Supervised Attention [13] (3) Edit Probability-Based Training [6] (4) Rectification [124] [125] Separately trained; passing cropped images: [7] Jointly trained; passing cropped feature maps”
• A Multi-Object Rectified Attention Network for Scene Text Recognition (2019, methodology): “Existing method [42] cannot capture the text shape in details if the width of the image is large.”
• An End-to-End TextSpotter with Explicit Alignment and Attention (2018, positive): “in [31], and spatial transformer networks [17] was incorporated for automatic rectification [32].”
• What Is Wrong With Scene Text Recognition Model Comparisons? Dataset and Model Analysis (2019, methodology): “(V) after the feature extraction stage [23, 24, 4].”
• Structured Multimodal Attentions for TextVQA (2020, background): “Most works address it separately by designing high performance text detector [31], [32], [33], [34] and sophisticated word recognizer [35], [36], [37], [38], [39], which can be combined together to obtain text contents in an image finally.”
• TextNet: Irregular Text Reading from Images with an End-to-End Trainable Network (2018, background): “Most recently, with the neural machine translation break-through by attention mechanism [1], several attention-based text recognition models [39][24] are developed to encode convolutional image features into sequence context with RNN, and predict each character using attention-based decoder.”
• FOTS: Fast Oriented Text Spotting with a Unified Network (2018, methodology): “[33] propose an end-to-end text spotting method, which uses a text proposal network inspired by RPN [41] for text detection and LSTM with attention mechanism [38, 45, 3] for text recognition.”
• Decoupled Attention Network for Text Recognition (2019, methodology): “Focusing on recognition of irregular text, (Shi et al. 2016), (Luo, Jin, and Sun 2019) and (Zhan and Lu 2019) proposed to rectify text distortion and recognize the rectified text with an attention-based recognizer; (Liu, Chen, and Wong 2018) proposed to rectify text at the character level; (Yang et…”
• Towards End-to-End Text Spotting with Convolutional Recurrent Neural Networks (2017, methodology): “[24] propose to recognize text using an attention-based sequence-to-sequence learning structure.”
• Multi-oriented Scene Text Detection via Corner Localization and Region Segmentation (2018, background): “Scene text detection, which aims at locating text in natural images, plays an important role in various text reading systems [35, 11, 50, 6, 21, 52, 5, 42, 41, 14, 8, 26].”