MaskOCR: Text Recognition with Masked Encoder-Decoder Pretraining

By Pengyuan Lyu, Chengquan Zhang, Shanshan Liu, Meina Qiao, Yangliu Xu, Liang Wu, Kun Yao, Junyu Han, Errui Ding, Jingdong Wang. 2022

51 citations recorded by the source corpus.

Text images contain both visual and linguistic information. However, existing pre-training techniques for text recognition mainly focus on either visual representation learning or linguistic knowledge learning. In this paper, we propose a novel approach MaskOCR to unify vision and language pre-training in the classical encoder-decoder recognition framework. We adopt the masked image modeling approach to pre-train the feature encoder using a large set of unlabeled real text images, which allows us to learn strong visual representations. In contrast to introducing linguistic knowledge with an additional language model, we directly pre-train the sequence decoder. Specifically, we transform text data into synthesized text images to unify the data modalities of vision and language, and enhance the language modeling capability of the sequence decoder using a proposed masked image-language modeling scheme. Significantly, the encoder is frozen during the pre-training phase of the sequence decoder. Experimental results demonstrate that our proposed method achieves superior performance on benchmark datasets, including Chinese and English text images.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Zhe Gan
  2. Lijuan Wang
  3. Zicheng Liu
  4. Jianfeng Wang
  5. Linjie Li
  6. Xiaowei Hu
  7. Zhengyuan Yang
  8. Ce Liu

In Computer Vision and Pattern Recognition

  1. Zhe Gan
  2. Lijuan Wang
  3. Zicheng Liu
  4. Jianfeng Wang
  5. Linjie Li
  6. Luc Van Gool
  7. Xiaowei Hu
  8. Cordelia Schmid

In Natural Language Processing

  1. Linjie Li
  2. Zhe Gan
  3. Kevin Qinghong Lin
  4. Ce Liu
  5. Jianfeng Wang
  6. Zicheng Liu
  7. Xiaowei Hu
  8. Zhengyuan Yang

Citations in the corpus