ESIR: End-To-End Scene Text Recognition via Iterative Image Rectification

By Fangneng Zhan, Shijian Lu. 2018

263 citations recorded by the source corpus.

Automated recognition of texts in scenes has been a research challenge for years, largely due to the arbitrary text appearance variation in perspective distortion, text line curvature, text styles and different types of imaging artifacts. The recent deep networks are capable of learning robust representations with respect to imaging artifacts and text style changes, but still face various problems while dealing with scene texts with perspective and curvature distortions. This paper presents an end-to-end trainable scene text recognition system (ESIR) that iteratively removes perspective distortion and text line curvature as driven by better scene text recognition performance. An innovative rectification network is developed, where a line-fitting transformation is designed to estimate the pose of text lines in scenes. Additionally, an iterative rectification framework is developed which corrects scene text distortions iteratively towards a fronto-parallel view. The ESIR is also robust to parameter initialization and easy to train, where the training needs only scene text images and word-level annotations as required by most scene text recognition systems. Extensive experiments over a number of public datasets show that the proposed ESIR is capable of rectifying scene text distortions accurately, achieving superior recognition performance for both normal scene text images and those suffering from perspective and curvature distortions.

Subfields: Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Xiang Bai
2. Cong Yao
3. Xin He
4. Shangbang Long
5. Lianwen Jin
6. Chengquan Zhang
7. Mingkun Yang
8. Errui Ding

In Media Technology

1. Cong Yao
2. Xiang Bai
3. Xin He
4. Shangbang Long
5. Lianwen Jin
6. Chengquan Zhang
7. Mingkun Yang
8. Errui Ding

Citations in the corpus

• All You Need Is Boundary: Toward Arbitrary-Shaped Text Spotting (2019, background): “In some latest works (Zhan and Lu 2019; Yang et al. 2019; Luo, Jin, and Sun 2019), recognition network is a common attentional sequence-to-sequence network.”
• Scene Text Detection and Recognition: The Deep Learning Era (2018, background): “Zhan and Lu (2019) propose to perform rectification multiple times to gradu-”
• Towards Accurate Scene Text Recognition With Semantic Reasoning Networks (2020, methodology): “Most of the recent works have attempted to improve the performance of scene text recognition from the perspective of extracting more robust and effective visual features, such as upgrading the backbone networks [5, 21, 31], adding rectification modules [30, 31, 40, 44] and improving attention mechan”
• Decoupled Attention Network for Text Recognition (2019, background): “Focusing on recognition of irregular text, (Shi et al. 2016), (Luo, Jin, and Sun 2019) and (Zhan and Lu 2019) proposed to rectify text distortion and recognize the rectified text with an attention-based recognizer; (Liu, Chen, and Wong 2018) proposed to rectify text at the character level; (Yang et…”
• TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models (2021, methodology): “, 2018) is applied to find a smooth spline interpolation between a set of fiducial points and normalize the text region to a predefined rectangle, while (Zhan and Lu, 2019) proposed an iterative rectification network to model the middle line of scene texts as well as the orientation and boundary of ”
• RobustScanner: Dynamically Enhancing Positional Clues for Robust Text Recognition (2020, methodology): “STN [17] was first introduced into text recognition network by Shi et al [40], which was extended by considering more flexible Transformation Thin-Plate-Spline (TPS) in [41,53] and conducting rectification iteratively in [55].”