Towards End-to-End Text Spotting with Convolutional Recurrent Neural Networks

By Hui Li, Peng Wang, Chunhua Shen. 2017

216 citations recorded by the source corpus.

In this work, we jointly address the problem of text detection and recognition in natural scene images based on convolutional recurrent neural networks. We propose a unified network that simultaneously localizes and recognizes text with a single forward pass, avoiding intermediate processes, such as image cropping, feature re-calculation, word separation, and character grouping. In contrast to existing approaches that consider text detection and recognition as two distinct tasks and tackle them one by one, the proposed framework settles these two tasks concurrently. The whole framework can be trained end-to-end, requiring only images, ground-truth bounding boxes and text labels. The convolutional features are calculated only once and shared by both detection and recognition, which saves processing time. Through multi-task training, the learned features become more informative and improves the overall performance. Our proposed method has achieved competitive performance on several benchmark datasets.

Subfields: Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Chunhua Shen
2. Xiang Bai
3. Dacheng Tao
4. Luc Van Gool
5. Yu Qiao
6. Xiaogang Wang
7. Cong Yao
8. Trevor Darrell

In Media Technology

1. Cong Yao
2. Xiang Bai
3. Minghui Liao
4. Xin He
5. Shangbang Long
6. Lianwen Jin
7. Pengyuan Lyu
8. Chunhua Shen

Citations in the corpus

• All You Need Is Boundary: Toward Arbitrary-Shaped Text Spotting (2019, methodology): “(Liu et al. 2018; Li, Wang, and Shen 2017; He et al. 2018; Busta et al. 2017) share a common idea: text region features are extracted for subsequent recognizer.”
• TextCaps: a Dataset for Image Captioning with Reading Comprehension (2020, background): “o understand the importance of OCR token, their semantic meaning, as well as relationship to visual context and other OCR tokens. Recent OCR models have shown reliability and performance improvements [8,32,21,23,15]. However, in our experiments we observe that OCR is far from a solved problem in rea”
• Empowering Things With Intelligence: A Survey of the Progress, Challenges, and Opportunities in Artificial Intelligence of Things (2020, background): “Recently, end-toend modeling of text detection and recognition has achieved impressive performance [90], [91].”
• Show, Attend and Read: A Simple and Strong Baseline for Irregular Text Recognition (2018, background): “(Li, Wang, and Shen 2017) combined a Faster-RCNN based text detector and a 1 D attention based recognizer into an end-to-end trainable system.”
• Mask TextSpotter: An End-to-End Trainable Neural Network for Spotting Text with Arbitrary Shapes (2018, methodology): “The left presents horizontal text spotting methods [30, 27]; The middle indicates oriented text spotting methods [3]; The right is our proposed method.”
• Multimodal Attention with Image Text Spatial Relationship for OCR-Based Image Captioning (2020, methodology): “Recently, end to end text spotting methods are gradually becoming mainstream [12, 19, 23].”
• Multi-oriented Scene Text Detection via Corner Localization and Region Segmentation (2018, background): “Scene text detection, which aims at locating text in natural images, plays an important role in various text reading systems [35, 11, 50, 6, 21, 52, 5, 42, 41, 14, 8, 26].”
• Detecting Curve Text in the Wild: New Dataset and New Solution (2017, background): “[20] also shows training with recognition can be conducive to text detection.”
• FOTS: Fast Oriented Text Spotting with a Unified Network (2018, methodology): “[33] propose an end-to-end text spotting method, which uses a text proposal network inspired by RPN [41] for text detection and LSTM with attention mechanism [38, 45, 3] for text recognition.”
• An End-to-End TextSpotter with Explicit Alignment and Attention (2018, background): “[23] proposed a unified network that simultaneously localizes and recognizes text in one forward pass by sharing convolution features under a curriculum strategy.”
• ASTER: An Attentional Scene Text Recognizer with Flexible Rectification (2019, background): “[37] integrated a text proposal network and an attentional sequence recognition network.”
• Scene Text Detection and Recognition: The Deep Learning Era (2018, methodology): “Recently, more end-to-end trainable networks are proposed to tackle the this problem [7] 14 , [11] 15 , [45], [72], [81].”
• Rotation-Sensitive Regression for Oriented Scene Text Detection (2018, background): “Recently, numerous inspiring ideas and promising methods [48, 49, 43, 19, 50, 11, 44, 26, 51, 38, 29, 13, 12, 14, 46, 5, 24, 42] have been proposed.”
• TextNet: Irregular Text Reading from Images with an End-to-End Trainable Network (2018, methodology): “ image segmentation accuracy, RoI-Align introduced by Mask R-CNN [13] aims to solve the misalignment between input and output features. To better design for text regions, the varying-size RoI pooling [25] has been developed to keep the aspect ratio unchanged for text recognition. Moreover, to handle”
• Structured Multimodal Attentions for TextVQA (2020, background): “Some recent works are also proposed to address both text detection and recognition in one framework [40], [41], [42], [43].”
• ABCNet: Real-Time Scene Text Spotting With Adaptive Bezier-Curve Network (2020, background): “, RoI Pooling [19], RoIRotate [25], Text-Align-Sampling [11], or RoI Transform [37] cannot properly align features of oriented or curved text (RoISlide [44] numerous predicting segments).”