An End-to-End Trainable Neural Network for Image-Based Sequence Recognition and Its Application to Scene Text Recognition

By Baoguang Shi, Xiang Bai, Cong Yao. 2015

2,937 citations recorded by the source corpus.

Image-based sequence recognition has been a long-standing research topic in computer vision. In this paper, we investigate the problem of scene text recognition, which is among the most important and challenging tasks in image-based sequence recognition. A novel neural network architecture, which integrates feature extraction, sequence modeling and transcription into a unified framework, is proposed. Compared with previous systems for scene text recognition, the proposed architecture possesses four distinctive properties: (1) It is end-to-end trainable, in contrast to most of the existing algorithms whose components are separately trained and tuned. (2) It naturally handles sequences in arbitrary lengths, involving no character segmentation or horizontal scale normalization. (3) It is not confined to any predefined lexicon and achieves remarkable performances in both lexicon-free and lexicon-based scene text recognition tasks. (4) It generates an effective yet much smaller model, which is more practical for real-world application scenarios. The experiments on standard benchmarks, including the IIIT-5K, Street View Text and ICDAR datasets, demonstrate the superiority of the proposed algorithm over the prior arts. Moreover, the proposed algorithm performs well in the task of image-based music score recognition, which evidently verifies the generality of it.

Subfields: Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

All You Need Is Boundary: Toward Arbitrary-Shaped Text Spotting (2019, methodology): “CRNN (Shi, Bai, and Yao 2017) is the ﬁrst method to treat text recognition as a sequence-to-sequence problem by combining CNN and RNN in an end-to-end network.”
WENETSPEECH: A 10000+ Hours Multi-Domain Mandarin Corpus for Speech Recognition (2021, methodology): “Text Recognition: a CRNN-CTC [26] based text recognition approach is used to recognize the detected subtitle area.”
MaskOCR: Text Recognition with Masked Encoder-Decoder Pretraining (2022, methodology): “In [12, 20, 30, 36, 44, 45, 60], the recognition model is pre-trained on synthesized data using supervised learning.”
Deep TextSpotter: An End-to-End Trainable Scene Text Localization and Recognition Framework (2017, methodology): “[24] train a fully-convolutional network with a bidirectional LSTM using the Connectionist Temporal Classification (CTC), which was first introduced by Graves et al.”
Towards End-to-End Text Spotting with Convolutional Recurrent Neural Networks (2017, methodology): “CRNN is then followed to recognize the detected bounding boxes.”
Robust Scene Text Recognition with Automatic Rectification (2016, methodology): “Overall, the model takes an input image I and outputs a sequence l = (l1, . . . , lT ), where lt is the t-th character, T is the variable string length.”
Towards Accurate Scene Text Recognition With Semantic Reasoning Networks (2020, methodology): “CRNN [29] firstly combined CNN and RNN to extract sequential visual features of a given text image, and then directly fed them into a CTC decoder to predict the best character category of each time step, where CTC [8] only maximized the probability of all the paths that can reach the ground truth ac”
Focusing Attention: Towards Accurate Text Recognition in Natural Images (2017, background): “Scene text recognition has attracted much research interest of the computer vision community [11, 21, 24].”
TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models (2021, methodology): “(Shi, Bai, and Yao 2016) proposed the standard CRNN, an end-to-end architecture combined by CNN and RNN.”
OCR-Free Document Understanding Transformer (2021, background): “Many modern text recognizer share a similar approach [37,53,52,59] that can be interpreted into a combination of several common deep modules [3].”
FOTS: Fast Oriented Text Spotting with a Unified Network (2018, methodology): “It is worth mentioning that in ICDAR 2015 text spotting task, our method outperforms previous best method [43, 44] by more than 15% in terms of F-measure.”
Mask TextSpotter: An End-to-End Trainable Neural Network for Spotting Text with Arbitrary Shapes (2018, methodology): “They first use a scene text detector [21, 30, 29] to localize text instances and then use a text recognizer [20, 44] to obtain the recognized text.”
MOTR: End-to-End Multiple-Object Tracking with TRansformer (2021, background): “Predicting sequence via sequence-to-sequence (seq2seq) with encoder-decoder architecture is popular in machine translation [30,31] and text recognition [25].”
TextBoxes++: A Single-Shot Oriented Scene Text Detector (2018, methodology): “For that, we adopt a stateof-the-art text recognizer called CRNN [12], which directly”
COFAR: Commonsense and Factual Reasoning in Image Search (2022, methodology): “We obtain these likelihood scores by using off-the-shelf approaches such as CRAFT+CRNN (Baek et al., 2019; Shi et al., 2017) for detecting and recognizing business brand mentions in the image, VGG face (Parkhi et al.”