PubLayNet: Largest Dataset Ever for Document Layout Analysis

By Xu Zhong, Jianbin Tang, Antonio Jimeno-Yepes. IEEE International Conference on Document Analysis and Recognition, 2019

638 citations recorded by the source corpus.

Recognizing the layout of unstructured digital documents is an important step when parsing the documents into structured machine-readable format for downstream applications. Deep neural networks that are developed for computer vision have been proven to be an effective method to analyze layout of document images. However, document layout datasets that are currently publicly available are several magnitudes smaller than established computing vision datasets. Models have to be trained by transfer learning from a base model that is pre-trained on a traditional computer vision dataset. In this paper, we develop the PubLayNet dataset for document layout analysis by automatically matching the XML representations and the content of over 1 million PDF articles that are publicly available on PubMed Central. The size of the dataset is comparable to established computer vision datasets, containing over 360 thousand document images, where typical document layout elements are annotated. The experiments demonstrate that deep neural networks trained on PubLayNet accurately recognize the layout of scientific articles. The pre-trained models are also a more effective base mode for transfer learning on a different document domain. We release the dataset (https://github.com/ibm-aur-nlp/PubLayNet) to support development and evaluation of more advanced models for document layout analysis.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/b5799d10df17de3232540e990da69553800d6376

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

LayoutLM: Pre-training of Text and Layout for Document Image Understanding (2019, methodology): “h as tabular areas. Hao et al. (2016) ﬁrst proposed a table detection method for PDF documents based on Convolutional Neural Networks (CNN). After that, (Schreiber et al., 2017; Soto & Yoo, 2019; Zhong et al., 2019) also leveraged more advanced Faster R-CNN model (Ren et al., 2015) or Mask R-CNN”
DocFormer: End-to-End Transformer for Document Understanding (2021, background): “Document understanding methods in the literature have used various combinations of image, spatial and text features in order to understand and extract information from structurally rich documents such as forms [19, 59, 13], tables [46, 58, 25], receipts [28, 27] and invoices [36, 44, 39].”
InfographicVQA (2021, methodology): “To this end, we use two pretrained object detection models — a Faster-RCNN [42] trained on Visual Genome [29] and a Mask-RCNN [19] trained on document images in PubLayNet [56] for Document Layout Analysis (DLA).”
LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking (2022, methodology): “LayoutLMv3 outperforms the compared ResNets [13, 51] and vision Transformer [27] backbones.”
LayoutLMv2: Multi-modal Pre-training for Visually-rich Document Understanding (2020, methodology): “For the ResNeXt-FPN part in the visual embedding layer, the backbone of a Mask-RCNN (He et al., 2017) model trained on Pub-LayNet (Zhong et al., 2019) is leveraged.”
Invoice Haystack: Benchmarking Document Retrieval and Visual Question Answering Under Strong Visual Homogeneity (2026, unknown): “Document AI has evolved from modular pipelines combining OCR [3,62], layout analysis [59, 77], and NLP models [18,45] toward unified end-to-end architectures.”