LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking

By Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, Furu Wei. ACM Multimedia, 2022

833 citations recorded by the source corpus.

Self-supervised pre-training techniques have achieved remarkable progress in Document AI. Most multimodal pre-trained models use a masked language modeling objective to learn bidirectional representations on the text modality, but they differ in pre-training objectives for the image modality. This discrepancy adds difficulty to multimodal representation learning. In this paper, we propose LayoutLMv3 to pre-train multimodal Transformers for Document AI with unified text and image masking. Additionally, LayoutLMv3 is pre-trained with a word-patch alignment objective to learn cross-modal alignment by predicting whether the corresponding image patch of a text word is masked. The simple unified architecture and training objectives make LayoutLMv3 a general-purpose pre-trained model for both text-centric and image-centric Document AI tasks. Experimental results show that LayoutLMv3 achieves state-of-the-art performance not only in text-centric tasks, including form understanding, receipt understanding, and document visual question answering, but also in image-centric tasks such as document image classification and document layout analysis. The code and models are publicly available at https://aka.ms/layoutlmv3.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/c689d1f3ae2447fd5b2f108b5b4436276e4d3761

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding (2022, methodology): “State-of-the-art document understanding models (Huang et al., 2022) therefore rely on the combination of (possibly noisy) OCR outputs with low resolution images.”
Argus-Retriever: Vision-LLM Late-Interaction Retrieval with Region-Aware Query-Conditioned MoE for Visual Document Retrieval (2026, unknown): “Pure text extraction can miss this information, and OCR-based pipelines often ignore the layout that makes the evidence interpretable (Huang et al., 2022).”
Invoice Haystack: Benchmarking Document Retrieval and Visual Question Answering Under Strong Visual Homogeneity (2026, unknown): “…catalyzing a paradigm shift from traditional modular pipelines comprising Optical Character Recognition (OCR), layout analysis, and rule-based information extraction [28, 71, 72] toward unified end-to-end architectures capable of simultaneous visual perception and semantic understanding [36, 39].”
Nemotron ColEmbed V2: Top-Performing Late Interaction embedding models for Visual Document Retrieval (2026, unknown): “Early systems [23, 24] relied on OCR-centric pipelines, where text was first extracted and then processed by layout-aware encoders that fused content with 2D positions.”