An End-to-End TextSpotter with Explicit Alignment and Attention

By Tong He, Zhi Tian, Weilin Huang, Chunhua Shen, Yu Qiao, Changming Sun. 2018

226 citations recorded by the source corpus.

Text detection and recognition in natural images have long been considered as two separate tasks that are processed sequentially. Jointly training two tasks is non-trivial due to significant differences in learning difficulties and convergence rates. In this work, we present a conceptually simple yet efficient framework that simultaneously processes the two tasks in a united framework. Our main contributions are three-fold: (1) we propose a novel text-alignment layer that allows it to precisely compute convolutional features of a text instance in arbitrary orientation, which is the key to boost the performance; (2) a character attention mechanism is introduced by using character spatial information as explicit supervision, leading to large improvements in recognition; (3) two technologies, together with a new RNN branch for word recognition, are integrated seamlessly into a single model which is end-to-end trainable. This allows the two tasks to work collaboratively by sharing convolutional features, which is critical to identify challenging text instances. Our model obtains impressive results in end-to-end recognition on the ICDAR 2015 [19], significantly advancing the most recent results [2], with improvements of F-measure from (0.54, 0.51, 0.47) to (0.82, 0.77, 0.63), by using a strong, weak and generic lexicon respectively. Thanks to joint training, our method can also serve as a good detector by achieving a new state-of-the-art detection performance on related benchmarks. Code is available at https://github.com/tonghe90/textspotter.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

All You Need Is Boundary: Toward Arbitrary-Shaped Text Spotting (2019, methodology): “Methods in (Liu et al. 2018; He et al. 2018; Busta et al. 2017; Sun et al. 2018) extract text features in the minimum oriented rectangular box, which can tackle oriented text.”
TextCaps: a Dataset for Image Captioning with Reading Comprehension (2020, background): “Recent OCR models have shown reliability and performance improvements [8,34,22,24,16].”
Scene Text Visual Question Answering (2019, methodology): “Scene Image OCR: A state of the art text recognition model [20] is used to process the test set images.”
Deep Learning for Generic Object Detection: A Survey (2018, background): “proposals by sharing full image convolutional features with the detection network. Most widely used object proposal method. Greatly improved the detection speed. DeepProposal[79] VGG16 Fast RCNN 0:74 (100) 0.92 (1000) 0:58 (100) 0.80 (1000) 0:12 (100) 0.16 (1000) 53:2 (100) (07) ICCV15 Generate prop”
Scene Text Detection and Recognition: The Deep Learning Era (2018, background): “Two-Stage Pipelines Recently, end-to-end trainable networks are proposed to tackle this problem (Bartz et al. 2017; Busta et al. 2017; Li et al. 2017a; He et al. 2018; Liu et al. 2018c), where feature maps instead of images are cropped and fed to recognition modules.”
Multi-Modal Graph Neural Network for Joint Reasoning on Vision and Scene Text (2020, methodology): “For experiments on ST-VQA dataset, due to no available OCR results are provided, we use TextSpotter [18] to extract scene text in images.”
Oriented R-CNN for Object Detection (2021, background): “In addition, some works [12, 28, 39, 27, 43, 16, 40, 47, 15] have explored one-stage or anchor-free oriented object detection frameworks: outputting object classes and oriented bounding boxes without region proposal generation and RoI alignment operation.”
TextNet: Irregular Text Reading from Images with an End-to-End Trainable Network (2018, background): “cropped text image to text sequence [38][24]. More recently, in contrast to multi-stage text reading systems, fully trainable models to tackle end-toend text reading task have been proposed in [25][5][16]. These approaches are capable of predicting locations and content of regular text in one model.”
ESIR: End-To-End Scene Text Recognition via Iterative Image Rectification (2018, background): “[14] designs a novel character attention mechanism for end-to-end scene text spotting.”
Multimodal Attention with Image Text Spatial Relationship for OCR-Based Image Captioning (2020, background): “Recently, end to end text spotting methods are gradually becoming mainstream [12, 19, 23].”
Mask TextSpotter: An End-to-End Trainable Neural Network for Spotting Text with Arbitrary Shapes (2018, methodology): “Recently, some researchers [7], [26], [39], [46] start to combine text detection and recognition with an end-to-end trainable network, which consists of two sub-models: a detection network for extracting text instances, and a sequenceto-sequence network for predicting the sequential labels of each t”
FCOS: A Simple and Strong Anchor-Free Object Detector (2020, methodology): “Besides, more signiﬁcantly, these methods are mainly used in special domain objection detection such as scene text detection [16], [53] or face detection [18], [50], since it is believed that these methods do not work well when applied to generic object detection with highly overlapped bounding…”