PaLI: A Jointly-Scaled Multilingual Language-Image Model

By Xi Chen, Xiao Wang, Soravit Changpinyo, A. Piergiovanni, Piotr Padlewski, Daniel M. Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer and 19 more. 2022

1,015 citations recorded by the source corpus.

Effective scaling and a flexible task interface enable large language models to excel at many tasks. We present PaLI (Pathways Language and Image model), a model that extends this approach to the joint modeling of language and vision. PaLI generates text based on visual and textual inputs, and with this interface performs many vision, language, and multimodal tasks, in many languages. To train PaLI, we make use of large pre-trained encoder-decoder language models and Vision Transformers (ViTs). This allows us to capitalize on their existing capabilities and leverage the substantial cost of training them. We find that joint scaling of the vision and language components is important. Since existing Transformers for language are much larger than their vision counterparts, we train a large, 4-billion parameter ViT (ViT-e) to quantify the benefits from even larger-capacity vision models. To train PaLI, we create a large multilingual mix of pretraining tasks, based on a new image-text training set containing 10B images and texts in over 100 languages. PaLI achieves state-of-the-art in multiple vision and language tasks (such as captioning, visual question-answering, scene-text understanding), while retaining a simple, modular, and scalable design.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Multimodal Learning

Citations in the corpus

PaLM-E: An Embodied Multimodal Language Model (2023, methodology): “Model Precision Recall F1-score PaLI (Zero-shot) (Chen et al., 2022) 0.59 0.98 0.73 CLIP-FT (Xiao et al., 2022) 0.50 0.95 0.65 CLIP-FT-hindsight (Xiao et al., 2022) 1”
Language Is Not All You Need: Aligning Perception with Language Models (2023, background): “In recent years, vision-language learning and representation models has garnered significant attention [2, 3, 4, 34, 50, 51, 52, 53, 54].”
Scaling Open-Vocabulary Object Detection (2023, methodology): “WebLI is a large dataset of images and texts available on the public Web.”
OpenVLA: An Open-Source Vision-Language-Action Model (2024, methodology): “VLAs provide a direct instantiation of using pretrained vision-and-language foundation models for robotics, directly fine-tuning visually-conditioned language models (VLMs) such as PaLI [19, 20] to generate robot control actions.”
SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities (2024, methodology): “PaLI [13].”
A Survey on Vision--Language--Action Models for Embodied AI (2024, neutral): “For a more comprehensive comparison of human datasets, we refer interested readers to [217].”
Learning Interactive Real-World Simulators (2023, methodology): “We finetune the PaLI 3B vision-language model [38] to predict low-level control actions (joint movements in ∆ x, ∆ y ) from an image observation and a task description (e.g., “move the blue cube to the right”) using behavioral cloning (BC) to serve as the low-level control policy and the baseline,…”
Position: Levels of AGI for Operationalizing Progress on the Path to AGI (2023, neutral): “In the next section, we introduce a level-based ontology of AGI.”
PromptCap: Prompt-Guided Task-Aware Image Captioning (2022, background): “On the vision and vision-language side, pretrained vision models [9,19,66,79] are able to recognize and describe visual contents, including challenging tiny visual objects, diverse open-world concepts, or scene text in images.”
Multimodal Large Language Models: A Survey (2023, methodology): “Recent research like BLIP-2, KOSMOS-1, BEiT-3, and PaLI [76] has highlighted two feasible approaches to address this issue: i) avoid catastrophic forgetting by using smaller networks and retraining from scratch with new data; ii) circumvent catastrophic forgetting by employing larger language…”
SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics (2025, neutral): “This design choice also differs from standard VLM architectures, where each decoder block typically includes both SA and CA layers (Laurençon et al., 2023; Alayrac et al., 2022; Chen et al., 2022).”
From Sparse to Soft Mixtures of Experts (2023, methodology): “For contrastive learning we train on WebLI (Chen et al., 2022), a proprietary dataset consisting of 10B images and alt-texts.”
Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning (2024, background): “In terms of training strategy, recent works [34, 3, 60, 80, 7, 38] commonly employ a 2-stage framework.”
Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding (2022, methodology): “Natural image understanding Pix2Seq uses the image-to-text architecture for core vision tasks such as object detection and instance segmentation (Chen et al., 2022b; 2021).”