DocFormer: End-to-End Transformer for Document Understanding

By Srikar Appalaraju, Bhavan A. Jasani, Bhargava Urala Kota, Yusheng Xie, Raghavan Manmatha. IEEE International Conference on Computer Vision, 2021

414 citations recorded by the source corpus.

We present DocFormer - a multi-modal transformer based architecture for the task of Visual Document Understanding (VDU). VDU is a challenging problem which aims to understand documents in their varied formats (forms, receipts etc.) and layouts. In addition, DocFormer is pre-trained in an unsupervised fashion using carefully designed tasks which encourage multi-modal interaction. DocFormer uses text, vision and spatial features and combines them using a novel multi-modal self-attention layer. DocFormer also shares learned spatial embeddings across modalities which makes it easy for the model to correlate text to visual tokens and vice versa. DocFormer is evaluated on 4 different datasets each with strong baselines. DocFormer achieves state-of-the-art results on all of them, sometimes beating models 4x its size (in no. of parameters).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/e6cbb6aa65d7761fabb6c5eb363f66524abe0a4c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

OCR-Free Document Understanding Transformer (2021, methodology): “Although the idea is simple, the methods showed remarkable performance improvements and became a main trend in recent years [62,35,2].”
LaTr: Layout-Aware Transformer for Scene-Text VQA (2021, background): “Recent document understanding literature [5, 68, 69] prove the value of layout information 1 https://huggingface.co/transformers/model_doc/ t5.html when working with Transformers.”
Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding (2022, background): “Document understanding State-of-the-art models in this domain are based on a pipeline of an external OCR sys-tem and a model that combines images and OCR annotations (Appalaraju et al., 2021; Powalski et al., 2021; Xu et al., 2021), inter alia .”
A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis (2023, methodology): “Prior works employ layout-informed tokens [80] or even multimodal tokens from visual inputs [6, 34, 36].”
MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge (2022, background): “Another promising future direction is to apply document understanding models such as LayoutLM [138, 137] and DocFormer [9] to learn actionable knowledge from these unstructured Wiki data.”
InfographicVQA (2021, methodology): “Models such as LayoutLMv2 [47], TILT [34], DocFormer [6] and StrucText [28] build on transformer-based architectures and leverage large-scale pretraining on unlabelled data, using pretraining objectives specifically designed for document understanding.”
Multimodal Learning With Transformers: A Survey (2022, background): “Scope This survey will discuss the multimodality specific designs of Transformer architecture including, but not limited to, the following modalities: RGB image [5], depth image [13], video [7], audio/speech/music [13], [14], [15], table [16], scene graph/layout [17], [18], [19], pose skeleton [20],”