Mask TextSpotter: An End-to-End Trainable Neural Network for Spotting Text with Arbitrary Shapes

By Pengyuan Lyu, Minghui Liao, Cong Yao, Xiang Bai. 2018

642 citations recorded by the source corpus.

Unifying text detection and text recognition in an end-to-end training fashion has become a new trend for reading text in the wild, as these two tasks are highly relevant and complementary. In this paper, we investigate the problem of scene text spotting, which aims at simultaneous text detection and recognition in natural images. An end-to-end trainable neural network named as Mask TextSpotter is presented. Different from the previous text spotters that follow the pipeline consisting of a proposal generation network and a sequence-to-sequence recognition network, Mask TextSpotter enjoys a simple and smooth end-to-end learning procedure, in which both detection and recognition can be achieved directly from two-dimensional space via semantic segmentation. Further, a spatial attention module is proposed to enhance the performance and universality. Benefiting from the proposed two-dimensional representation on both detection and recognition, it easily handles text instances of irregular shapes, for instance, curved text. We evaluate it on four English datasets and one multi-language dataset, achieving consistently superior performance over state-of-the-art methods in both detection and end-to-end text recognition tasks. Moreover, we further investigate the recognition module of our method separately, which significantly outperforms state-of-the-art methods on both regular and irregular text datasets for scene text recognition.

Subfields: Computer Vision and Pattern Recognition, Media Technology, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

In Human-Computer Interaction

Citations in the corpus

GIT: A Generative Image-to-text Transformer for Vision and Language (2022, methodology): “Method FT data Average SAM (Liao et al., 2019) MJ+ST 87.”
Scene Text Visual Question Answering (2019, background): “Later works focus towards end-to-end architectures such as the ones presented by [8, 39, 20], which mostly con-”
Mix-review: Alleviate Forgetting in the Pretrain-Finetune Framework for Neural Language Generation Models (2019, positive): “Mix-review can also be viewed as a form of multi-task learning (Li et al., 2019), which has been shown to be useful in neural machine translation (NMT) (Niehues & Cho, 2017), speech recognition (Toshniwal et al., 2017), optical character recognition (OCR) (Liao et al., 2019), etc.”
Real-Time Scene Text Detection With Differentiable Binarization and Adaptive Scale Fusion (2022, methodology): “Mask TextSpotter [23], [38] detected arbitrary-shape text instances in an instance segmentationmanner based onMask R-CNN [13].”
ABCNet: Real-Time Scene Text Spotting With Adaptive Bezier-Curve Network (2020, methodology): “Recently, many end-to-end methods [31, 37, 34, 24, 44, 21] have significantly improved the performance of oriented or curved scene text spotting.”
Total-Text: toward orientation robustness in scene text detection (2019, background): “Since the publication of [12], several works have attempted to address the curved text detection problem [13,44,45,52].”
Scene Text Detection and Recognition: The Deep Learning Era (2018, methodology): “Similarly, Lyu et al. [88] modiﬁes the Mask R-CNN [42] framework, so that for each region of interest—in the form of axis-aligned rectangles—character masks are predicted solely for each type of alphabets.”
MPDIoU: A Loss for Efficient and Accurate Bounding Box Regression (2023, background): “A typical form of L is `n-norm, for example, mean-square error (MSE) loss and Smooth-`1 loss [43], which have been widely adopted in object detection [44]; pedestrian detection [45, 46]; scene text spotting [34, 47]; 3D object detection [48, 49]; pose estimation [50, 51]; and instance segmentation [”
Efficient and Accurate Arbitrary-Shaped Text Detection With Pixel Aggregation Network (2019, methodology): “Mask Text Spotter [36] and SPCNet [49] regard text detection as an instance segmentation problem and use Mask R-CNN [12] for arbitrary text detection.”
Real-time Scene Text Detection with Differentiable Binarization (2019, methodology): “Text border is used in (Xue, Lu, and Zhan 2018) to split the text instances, Mask TextSpotter (Lyu et al. 2018a; Liao et al. 2019) detected arbitrary-shape text instances in an instance segmentation manner based on Mask R-CNN.”
Character Region Awareness for Text Detection (2019, background): “Recently, scene text detectors based on deep learning have shown promising performance [8, 40, 21, 4, 11, 10, 12, 13, 17, 24, 25, 32, 26].”
MaskOCR: Text Recognition with Masked Encoder-Decoder Pretraining (2022, background): “Text recognition can be achieved through various approaches, including character-based [22, 31, 34, 52], word-based [36], and sequence-based [12,44,45,57,60] methods.”
Symmetry-Constrained Rectification Network for Scene Text Recognition (2019, background): “Scene text reading [60, 32, 59, 54, 53, 36, 35] is an important, active research area in computer vision, which can be applied to a wide range of real-world applications, such as self-driving cars, assistant position systems, and guide board recognition [43].”
Arbitrary Shape Scene Text Detection With Adaptive Text Region Representation (2019, methodology): “Moreover, Mask TextSpot-ter [20] which is inspired by Mask R-CNN, can handle text instances of irregular shapes via semantic segmentation.”