StructuralLM: Structural Pre-training for Form Understanding

By Chenliang Li, Bin Bi, Ming Yan, Wei Wang, Songfang Huang, Fei Huang, Luo Si. 2021

149 citations recorded by the source corpus.

Large pre-trained language models achieve state-of-the-art results when fine-tuned on downstream NLP tasks. However, they almost exclusively focus on text-only representation, while neglecting cell-level layout information that is important for form image understanding. In this paper, we propose a new pre-training approach, StructuralLM, to jointly leverage cell and layout information from scanned documents. Specifically, we pre-train StructuralLM with two new designs to make the most of the interactions of cell and layout information: 1) each cell as a semantic unit; 2) classification of cell positions. The pre-trained StructuralLM achieves new state-of-the-art results in different types of downstream tasks, including form understanding (from 78.95 to 85.14), document visual question answering (from 72.59 to 83.94) and document image classification (from 94.43 to 96.08).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Extraction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Furu Wei
2. Mohit Bansal
3. Luke S. Zettlemoyer
4. Jianfeng Gao
5. Quoc V. Le
6. Percy Liang
7. Yejin Choi
8. Hannaneh Hajishirzi

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Jianfeng Gao
3. Dacheng Tao
4. Hannaneh Hajishirzi
5. Trevor Darrell
6. Jitendra Malik
7. Quoc V. Le
8. Xiaogang Wang

In Information Extraction

1. Fei Huang
2. Ming-Wei Chang
3. Songfang Huang
4. Kenton C. T. Lee
5. Kristina Toutanova
6. Peter Shaw
7. Furu Wei

Citations in the corpus

• Achieving Human Parity on Visual Question Answering (2021, methodology): “In addition, StructuralLM is capable of learning the interactions between text and layout, which makes StructuralLM aware of location of text present in an image.”
• A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis (2023, background): “In the foreseeable future, this technology could lead to the development of sophisticated AI assistant tools for computers and smartphones, enhancing productivity and accessibility for society.”
• Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding (2022, background): “Screen2Words (Wang et al., 2021a) is an image captioning task where the input is an app screenshot and the caption describes the functionality of the page (see Figure 1 for an example).”
• Understanding HTML with Large Language Models (2022, methodology): “StructuralLM ((Li et al., 2021a)) is an approach specifically tailored for document understanding (i.e., combinations of images and text), and thus makes several simplifying assumptions for its model that limit its applicability to HTML understanding (i.e., trees of elements with a richer structure…”
• OCR-Free Document Understanding Transformer (2021, methodology): “Recently, with BERT [8], the methods based on a combination of CV and NLP were widely proposed [63,34].”
• LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking (2022, methodology): “As simply using linear image embedding has improved FUNSD, MIM does not further contribute to FUNSD.”