ASTER: An Attentional Scene Text Recognizer with Flexible Rectification

By Baoguang Shi, Mingkun Yang, Xinggang Wang, Pengyuan Lyu, Cong Yao, Xiang Bai. 2019

837 citations recorded by the source corpus.

A challenging aspect of scene text recognition is to handle text with distortions or irregular layout. In particular, perspective text and curved text are common in natural scenes and are difficult to recognize. In this work, we introduce ASTER, an end-to-end neural network model that comprises a rectification network and a recognition network. The rectification network adaptively transforms an input image into a new one, rectifying the text in it. It is powered by a flexible Thin-Plate Spline transformation which handles a variety of text irregularities and is trained without human annotations. The recognition network is an attentional sequence-to-sequence model that predicts a character sequence directly from the rectified image. The whole model is trained end to end, requiring only images and their groundtruth text. Through extensive experiments, we verify the effectiveness of the rectification and demonstrate the state-of-the-art recognition performance of ASTER. Furthermore, we demonstrate that ASTER is a powerful component in end-to-end recognition systems, for its ability to enhance the detector.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Xiang Bai
2. Tao A. Xiang
3. Chunhua Shen
4. Cong Yao
5. Yi-Zhe Song
6. Yu Qiao
7. Luc Van Gool
8. Dacheng Tao

Citations in the corpus

• All You Need Is Boundary: Toward Arbitrary-Shaped Text Spotting (2019, methodology): “Following (Shi et al. 2019), ArbitraryRoIAlign is adopted to flatten features of text instances with arbitrary shapes.”
• TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models (2021, methodology): “(Shi et al. 2016; Baek et al. 2019; Litman et al. 2020; Shi et al. 2018; Zhan and Lu 2019) addressed this problem by processing the input images with an initial rectification step.”
• Symmetry-Constrained Rectification Network for Scene Text Recognition (2019, methodology): “Deep learning based methods [20, 17, 25, 48, 45, 44, 7, 29, 3, 8, 46, 57] have been dominating this area in recent years.”
• Show, Attend and Read: A Simple and Strong Baseline for Irregular Text Recognition (2018, methodology): “Figure 1: The comparison of our proposed 2D attention based and the rectification based (Shi et al. 2018) irregular text recognizers.”
• Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting (2021, methodology): “Following [56], the (word) image encoder extends a conventional ResNet architecture with a 2-layer BLSTM image feature encoder before producing a final state that provides the latent vector for the input image.”
• Visual Semantics Allow for Textual Reasoning Better in Scene Text Recognition (2021, positive): “…recognition (VR) task and perform character-level recognition on input images, including visual-text sequence translation-based methods (Yang et al. 2017; Shi et al. 2018; Baek et al. 2019; Li et al. 2019; Litman et al. 2020) and semantic segmentation-based methods (Liao et al. 2019; Wan et 2020a).”
• Decoupled Attention Network for Text Recognition (2019, methodology): “We adopt a similar CNN-based feature encoder as previous study (Shi et al. 2018).”
• TextSnake: A Flexible Representation for Detecting Text of Arbitrary Shapes (2018, background): “Numerous innovative works [39, 29, 47, 17, 10, 46, 30, 36, 6, 9, 34, 22, 28, 40, 31] are proposed, achieving excellent performances on standard benchmarks.”
• Scene Text Detection and Recognition: The Deep Learning Era (2018, methodology): “(1) CNN+RNN [123] (2) Conv Sequence Learning [28] (3) Sliding Window [158] (1) Attention-based decoding [70] (2) Supervised Attention [13] (3) Edit Probability-Based Training [6] (4) Rectification [124] [125] Separately trained; passing cropped images: [7] Jointly trained; passing cropped feature ma”
• Towards Accurate Scene Text Recognition With Semantic Reasoning Networks (2020, methodology): “Most of the recent works have attempted to improve the performance of scene text recognition from the perspective of extracting more robust and effective visual features, such as upgrading the backbone networks [5, 21, 31], adding rectification modules [30, 31, 40, 44] and improving attention mechan”
• Mask TextSpotter: An End-to-End Trainable Neural Network for Spotting Text with Arbitrary Shapes (2018, background): “Scene text recognition [53, 46] aims at decoding the detected or cropped image regions into character sequences.”
• Language-Agnostic Visual Embeddings for Cross-Script Handwriting Retrieval (2026, unknown): “Transcription-based methods (OCR) decode textual content via modular pipelines [Liao et al. , 2017; Liao et al. , 2022; Shi et al. , 2018; Li et al. , 2021], evolving into high-performance tools [Li et al. , 2023; Team, 2021; Wei et al. , 2025].”