LayoutLMv2: Multi-modal Pre-training for Visually-rich Document Understanding

By Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei A. F. Florêncio, Cha Zhang, Wanxiang Che and 2 more. Annual Meeting of the Association for Computational Linguistics, 2020

702 citations recorded by the source corpus.

Pre-training of text and layout has proved effective in a variety of visually-rich document understanding tasks due to its effective model architecture and the advantage of large-scale unlabeled scanned/digital-born documents. We propose LayoutLMv2 architecture with new pre-training tasks to model the interaction among text, layout, and image in a single multi-modal framework. Specifically, with a two-stream multi-modal Transformer encoder, LayoutLMv2 uses not only the existing masked visual-language modeling task but also the new text-image alignment and text-image matching tasks, which make it better capture the cross-modality interaction in the pre-training stage. Meanwhile, it also integrates a spatial-aware self-attention mechanism into the Transformer architecture so that the model can fully understand the relative positional relationship among different text blocks. Experiment results show that LayoutLMv2 outperforms LayoutLM by a large margin and achieves new state-of-the-art results on a wide variety of downstream visually-rich document understanding tasks, including FUNSD (0.7895 to 0.8420), CORD (0.9493 to 0.9601), SROIE (0.9524 to 0.9781), Kleister-NDA (0.8340 to 0.8520), RVL-CDIP (0.9443 to 0.9564), and DocVQA (0.7295 to 0.8672).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/d42f77ee45e267cd0aa66103bbec8abf7f6d4767

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

OCR-Free Document Understanding Transformer (2021, methodology): “In particular, Donut surpasses the Lay-outLMv2 accuracy reported in [64], while using fewer parameters with the 2x faster speed.”
Multimodal Learning With Transformers: A Survey (2022, background): “language [27], sign language [28], [29], [30], point cloud [31], symbolic knowledge (graph) [32], [33], multimodal knowledge graph [34], sketch drawing [35], [36], [37], [38], 3D object/scene [39], [40], [41], document [42], [43], programming code [44] and Abstract Syntax Tree (AST) – a kind of grap”
Knowledge Graph Prompting for Multi-Document Question Answering (2023, methodology): “One common solution to overcome this limitation in conventional OD-QA and single document question-answering (D-QA) (Xu et al. 2020; Mathew, Karatzas, and Jawahar 2021) is to retrieve grounding contexts and derive faithful answers from the contexts, i.e., retrieve-and-read (Zhu et al. 2021; Ju et…”
LaTr: Layout-Aware Transformer for Scene-Text VQA (2021, background): “html 2-D Spatial Embedding Recent document understanding literature [5, 72, 73] prove the value of layout information when working with Transformers.”
LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking (2022, methodology): “Existing multimodal models in Document AI either extract CNN grid features [2, 49] or rely on an object detector like Faster R-CNN to extract region features [13, 28, 36, 47] for image embeddings, which accounts for heavy computation bottleneck or require region supervision.”
MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge (2022, background): “[121] Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, Min Zhang, and Lidong Zhou.”
DocFormer: End-to-End Transformer for Document Understanding (2021, methodology): “Transformers have recently been used for VDU [26, 56, 57].”
Invoice Haystack: Benchmarking Document Retrieval and Visual Question Answering Under Strong Visual Homogeneity (2026, unknown): “…catalyzing a paradigm shift from traditional modular pipelines comprising Optical Character Recognition (OCR), layout analysis, and rule-based information extraction [28, 71, 72] toward unified end-to-end architectures capable of simultaneous visual perception and semantic understanding [36, 39].”
VaRS-Doc: Interpretation-Aware Variant Representations via Latent Self-Probing for Visual Document Retrieval (2026, unknown): “Early document understanding meth-ods, such as LayoutLM (Xu et al. 2020) and LayoutLMv2 (Xu et al. 2021), incorporate OCR tokens and layout information to model document structure.”
LightSTAR: Efficient Visual Document Retrieval via Lightweight Selection with Vision-Adaptive Refinement (2026, unknown): “Early works such as LayoutLM [49] and LayoutLMv2 [48] incorporate layout information by combining visual features with text embeddings extracted from OCR.”