TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models
By Minghao Li, Tengchao Lv, Lei Cui, Yijuan Lu, Dinei A. F. Florêncio, Cha Zhang, Zhoujun Li, Furu Wei. 2021
682 citations recorded by the source corpus.
Text recognition is a long-standing research problem for document digitalization. Existing approaches are usually built based on CNN for image understanding and RNN for char-level text generation. In addition, another language model is usually needed to improve the overall accuracy as a post-processing step. In this paper, we propose an end-to-end text recognition approach with pre-trained image Transformer and text Transformer models, namely TrOCR, which leverages the Transformer architecture for both image understanding and wordpiece-level text generation. The TrOCR model is simple but effective, and can be pre-trained with large-scale synthetic data and fine-tuned with human-labeled datasets. Experiments show that the TrOCR model outperforms the current state-of-the-art models on the printed, handwritten and scene text recognition tasks. The TrOCR models and code are publicly available at https://aka.ms/trocr.
Subfields: Computer Vision and Pattern Recognition, Media Technology, Model Architectures
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Sergey Levine
- Trevor Darrell
- Jitendra Malik
- Li Fei-Fei
- Dacheng Tao
- Xiaogang Wang
- Ming-Hsuan Yang
In Media Technology
- Chengquan Zhang
- Pengyuan Lyu
- Errui Ding
- Junyu Han
- Tianhe Yu
- Jingdong Wang
- Sergey Levine
- Igor Mordatch
In Model Architectures
- Jingdong Wang
- Sergey Levine
- Pierre Sermanet
- Jonathan Tompson
- Corey Lynch
- Quoc V. Le
- Pengyuan Lyu
- Tianhe Yu
Citations in the corpus
- PaLM-E: An Embodied Multimodal Language Model (2023, background): “…VLMs are capable of simultaneously understanding both images and text, and can be applied to tasks such as visual question answering (Zhou et al., 2020; Zellers et al., 2021b), cap-tioning (Hu et al., 2022), optical character recognition (Li et al., 2021), and object detection (Chen et al., 2021b).”
- MaskOCR: Text Recognition with Masked Encoder-Decoder Pretraining (2022, methodology): “In [12, 20, 30, 36, 44, 45, 60], the recognition model is pre-trained on synthesized data using supervised learning.”
- Language-Agnostic Visual Embeddings for Cross-Script Handwriting Retrieval (2026, unknown): “Transcription-based methods (OCR) decode textual content via modular pipelines [Liao et al. , 2017; Liao et al. , 2022; Shi et al. , 2018; Li et al. , 2021], evolving into high-performance tools [Li et al. , 2023; Team, 2021; Wei et al. , 2025].”