Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition

By Shancheng Fang, Hongtao Xie, Yuxin Wang, Zhendong Mao, Yongdong Zhang. 2021

416 citations recorded by the source corpus.

Linguistic knowledge is of great benefit to scene text recognition. However, how to effectively model linguistic rules in end-to-end deep networks remains a research challenge. In this paper, we argue that the limited capacity of language models comes from: 1) implicitly language modeling; 2) unidirectional feature representation; and 3) language model with noise input. Correspondingly, we propose an autonomous, bidirectional and iterative ABINet for scene text recognition. Firstly, the autonomous suggests to block gradient flow between vision and language models to enforce explicitly language modeling. Secondly, a novel bidirectional cloze network (BCN) as the language model is proposed based on bidirectional feature representation. Thirdly, we propose an execution manner of iterative correction for language model which can effectively alleviate the impact of noise input. Additionally, based on the ensemble of iterative predictions, we propose a self-training method which can learn from unlabeled images effectively. Extensive experiments indicate that ABINet has superiority on low-quality images and achieves state-of-the-art results on several mainstream benchmarks. Besides, the ABINet trained with ensemble self-training shows promising improvement in realizing human-level recognition. Code is available at https://github.com/FangShancheng/ABINet.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

GIT: A Generative Image-to-text Transformer for Vision and Language (2022, methodology): “Recently, a growing interest is in dense representation (Huang et al., 2020; Wang et al., 2021b;a; Kim et al., 2021; Fang et al., 2021b; Dou et al., 2021; Li et al., 2021a) from the feature map, which requires no bounding box annotations.”
Visual Semantics Allow for Textual Reasoning Better in Scene Text Recognition (2021, neutral): “…STR methods, including CTC-based method ( e.g. , CRNN (Shi, Bai, and Yao 2016)), 1D attention-based method ( e.g. , TRBA (Baek et al. 2019)), 2D attention-based method ( e.g. , Base2D (Yan et al. 2021)), and transformer-based methods ( e.g. , SRN (Yu et al. 2020) and ABINet-LV (Fang et al. 2021)).”
LaTr: Layout-Aware Transformer for Scene-Text VQA (2021, background): “1 (b) we visualize the information extracted by the optical character recognition (OCR) system [1, 6, 13, 36], exhibiting three question categories: the ﬁrst type can be answered with just the text tokens; the second type can be answered with text and layout information ( right vs left ); the third…”
MaskOCR: Text Recognition with Masked Encoder-Decoder Pretraining (2022, methodology): “In [12, 20, 30, 36, 44, 45, 60], the recognition model is pre-trained on synthesized data using supervised learning.”