Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding

By Kenton C. T. Lee, Mandar Joshi, Iulia Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, Kristina Toutanova. International Conference on Machine Learning, 2022

473 citations recorded by the source corpus.

Visually-situated language is ubiquitous -- sources range from textbooks with diagrams to web pages with images and tables, to mobile apps with buttons and forms. Perhaps due to this diversity, previous work has typically relied on domain-specific recipes with limited sharing of the underlying data, model architectures, and objectives. We present Pix2Struct, a pretrained image-to-text model for purely visual language understanding, which can be finetuned on tasks containing visually-situated language. Pix2Struct is pretrained by learning to parse masked screenshots of web pages into simplified HTML. The web, with its richness of visual elements cleanly reflected in the HTML structure, provides a large source of pretraining data well suited to the diversity of downstream tasks. Intuitively, this objective subsumes common pretraining signals such as OCR, language modeling, image captioning. In addition to the novel pretraining strategy, we introduce a variable-resolution input representation and a more flexible integration of language and vision inputs, where language prompts such as questions are rendered directly on top of the input image. For the first time, we show that a single pretrained model can achieve state-of-the-art results in six out of nine tasks across four domains: documents, illustrations, user interfaces, and natural images.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/e1484706c0fab932fc9804df328044b3cb2f110d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

GPT-4V(ision) is a Generalist Web Agent, if Grounded (2024, methodology): “Pix2Act (Shaw et al., 2023) leverages Pix2Struct (Lee et al., 2022) to parse screenshot images into simplified HTML to complete GUI-based tasks (Shaw et al., 2023; Liu et al., 2018; Shi et al., 2017; Mazumder & Riva, 2020; Yao et al., 2022).”
WebArena: A Realistic Web Environment for Building Autonomous Agents (2023, background): “Adding a multi-modal dimension, Lee et al. (2023) and Shaw et al. (2023) develop agents that predict actions based on screenshots of web pages rather than relying on the text-based DOM trees.”
PaLI-X: On Scaling up a Multilingual Vision and Language Model (2023, methodology): “Being able to have the text choices as input benefits PaLI-X compared with the previous SoTA Pix2Struct [27] which has to render the text on the image, but this does not explain all the improvements.”
OS-ATLAS: A Foundation Action Model for Generalist GUI Agents (2024, background): “While some early efforts have focused on gathering screenshots corpus for websites (Lee et al., 2022; Chen et al., 2024b) Figure 1: (Left) The OS-Atlas model operates in three distinct modes to cater to various research needs.”
CogAgent: A Visual Language Model for GUI Agents (2023, unknown): “5 and Pix2Struct [16]).”
What matters when building vision-language models? (2024, methodology): “Following Lee et al. (2023); Dehghani et al. (2023), we pass the image patches to the vision encoder without resizing the image or modifying its aspect ratio.”
MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts (2023, methodology): “Prior work, such as PixStruct (Lee et al., 2023), MatCha (Liu et al.”
Invoice Haystack: Benchmarking Document Retrieval and Visual Question Answering Under Strong Visual Homogeneity (2026, unknown): “…catalyzing a paradigm shift from traditional modular pipelines comprising Optical Character Recognition (OCR), layout analysis, and rule-based information extraction [28, 71, 72] toward unified end-to-end architectures capable of simultaneous visual perception and semantic understanding [36, 39].”
VaRS-Doc: Interpretation-Aware Variant Representations via Latent Self-Probing for Visual Document Retrieval (2026, unknown): “Later OCR-free models, including Donut (Kim et al. 2022) and Pix2Struct (Lee et al. 2023), directly process document images for structured visual understanding (Guan et al. 2025).”
LightSTAR: Efficient Visual Document Retrieval via Lightweight Selection with Vision-Adaptive Refinement (2026, unknown): “More recent approaches adopt an OCR-free paradigm: Donut [28] employs an encoder-decoder architecture to directly generate text from document images, while Pix2Struct [30] pretrains on page screenshots for structured understanding.”