LayoutLM: Pre-training of Text and Layout for Document Image Understanding

By Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou. Knowledge Discovery and Data Mining, 2019

1,049 citations recorded by the source corpus.

Pre-training techniques have been verified successfully in a variety of NLP tasks in recent years. Despite the widespread use of pre-training models for NLP applications, they almost exclusively focus on text-level manipulation, while neglecting layout and style information that is vital for document image understanding. In this paper, we propose the LayoutLM to jointly model interactions between text and layout information across scanned document images, which is beneficial for a great number of real-world document image understanding tasks such as information extraction from scanned documents. Furthermore, we also leverage image features to incorporate words' visual information into LayoutLM. To the best of our knowledge, this is the first time that text and layout are jointly learned in a single framework for document-level pre-training. It achieves new state-of-the-art results in several downstream tasks, including form understanding (from 70.72 to 79.27), receipt understanding (from 94.02 to 95.24) and document image classification (from 93.07 to 94.42). The code and pre-trained LayoutLM models are publicly available at https://aka.ms/layoutlm.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/3465c06c872d8c48d628c5fc2d484087719351b6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Trevor Darrell
4. Andrew Zisserman
5. Ming-Hsuan Yang
6. Xiaogang Wang
7. Alan L. Yuille
8. Chunhua Shen

Citations in the corpus

• MathBERT: A Pre-Trained Model for Mathematical Formula Understanding (2021, methodology): “LayoutLM [Xu et al. , 2020] was proposed for document understanding, which was jointly trained on multi-modal information including text, image and layout.”
• MaskOCR: Text Recognition with Masked Encoder-Decoder Pretraining (2022, methodology): “For instance, the MVLM module in Lay-outLM [56] is proposed to align visual and textual representations, while VLPT [48] uses image-text contrastive learning to enhance visual representations.”
• Co-Scale Conv-Attentional Image Transformers (2021, background): “A notable recent development in artificial intelligence is the creation of attention mechanisms [38] and Transformers [31], which have made a profound impact in a range of fields including natural language processing [7, 20], document analysis [39], speech recognition [8], and computer vision [9, 3]”
• A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis (2023, methodology): “Prior works employ layout-informed tokens (Xu et al., 2019) or even multimodal tokens from visual inputs (Appalaraju et al.”
• InfographicVQA (2021, methodology): “LayoutLM [52] extends BERT by incorporating layout information into the original BERT model.”
• LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking (2022, methodology): “68* LayoutLMBASE [47] 160M T+L+I (R) ResNet-101 (fine-tune) 79.”
• LayoutLMv2: Multi-modal Pre-training for Visually-rich Document Understanding (2020, methodology): “In this paper, we present an improved version of LayoutLM (Xu et al., 2020), aka LayoutLMv2 .”
• StructuralLM: Structural Pre-training for Form Understanding (2021, methodology): “Our approach is inspired by LayoutLM(Xu et al., 2019), but different from it in three ways.”
• LaTr: Layout-Aware Transformer for Scene-Text VQA (2021, background): “Recent document understanding literature [5, 68, 69] prove the value of layout information 1 https://huggingface.co/transformers/model_doc/ t5.html when working with Transformers.”
• MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge (2022, background): “Another promising future direction is to apply document understanding models such as LayoutLM [138, 137] and DocFormer [9] to learn actionable knowledge from these unstructured Wiki data.”
• OCR-Free Document Understanding Transformer (2021, methodology): “‡ # parameters for OCR should be considered Fine-tuning set OCR #Params † Time (ms) ANLS test set ANLS ∗ handwritten BERT [64] train set ✓ 110M + α ‡ 1517 63.5 n/a LayoutLM[65] train set ✓ 113M + α ‡ 1519 69.8 n/a LayoutLMv2[64] train set errors is an option to strengthen the pipeline [51,50,10] or ”
• Invoice Haystack: Benchmarking Document Retrieval and Visual Question Answering Under Strong Visual Homogeneity (2026, unknown): “…catalyzing a paradigm shift from traditional modular pipelines comprising Optical Character Recognition (OCR), layout analysis, and rule-based information extraction [28, 71, 72] toward unified end-to-end architectures capable of simultaneous visual perception and semantic understanding [36, 39].”
• VaRS-Doc: Interpretation-Aware Variant Representations via Latent Self-Probing for Visual Document Retrieval (2026, unknown): “Early document understanding meth-ods, such as LayoutLM (Xu et al. 2020) and LayoutLMv2 (Xu et al. 2021), incorporate OCR tokens and layout information to model document structure.”
• LightSTAR: Efficient Visual Document Retrieval via Lightweight Selection with Vision-Adaptive Refinement (2026, unknown): “Early works such as LayoutLM [49] and LayoutLMv2 [48] incorporate layout information by combining visual features with text embeddings extracted from OCR.”
• Nemotron ColEmbed V2: Top-Performing Late Interaction embedding models for Visual Document Retrieval (2026, unknown): “Early systems [23, 24] relied on OCR-centric pipelines, where text was first extracted and then processed by layout-aware encoders that fused content with 2D positions.”