PaLI-X: On Scaling up a Multilingual Vision and Language Model

By Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz, Sebastian Goodman, Xiao Wang, Yi Tay and 33 more. 2023

301 citations recorded by the source corpus.

We present the training recipe and results of scaling up PaLI-X, a multilingual vision and language model, both in terms of size of the components and the breadth of its training task mixture. Our model achieves new levels of performance on a wide-range of varied and complex tasks, including multiple image-based captioning and question-answering tasks, image-based document understanding and few-shot (in-context) learning, as well as object detection, video question answering, and video captioning. PaLI-X advances the state-of-the-art on most vision-and-language benchmarks considered (25+ of them). Finally, we observe emerging capabilities, such as complex counting and multilingual object detection, tasks that are not explicitly in the training mix.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Multimodal Learning

Citations in the corpus

Language Is Not All You Need: Aligning Perception with Language Models (2023, background): “In recent years, vision-language learning and representation models has garnered significant attention [2, 3, 4, 34, 50, 51, 52, 53, 54].”
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control (2023, methodology): “…leverage pre-trained VLMs: (1) RT-2-PaLI-X is built from 5B and 55B PaLI-X [16], and (2) For training, we leverage the original web scale data from Chen et al. [16] and Driess et al. [17], which consists of visual question answering, captioning, and unstructured interwoven image and text examples.”
Otter: A Multi-Modal Model With In-Context Instruction Tuning (2023, unknown): “…which follows natural language instructions effectively to accomplish real-world tasks and allows for customizing task-speciﬁc rules into instructions during downstream ﬁne-tuning, enabling pre-trained models to comprehend user intents more effectively and produce accurate and relevant responses.”
CogAgent: A Visual Language Model for GUI Agents (2023, unknown): “It is a common problem among VLMs, e.g. LLaVA [21] and PALI-X [8] are pre-trained at a low resolution of 224 ⇥ 224 on the general domain.”
Learning Interactive Real-World Simulators (2023, methodology): “The history conditioned base model operates at temporal and spatial resolution [16 , 24 , 40] , and the two spatial super-resolution models operate at spatial resolution [24 , 40] → [48 , 80] and [48 , 80] → [192 , 320] , respectively.”
What matters when building vision-language models? (2024, unknown): “We fine-tune Idefics2 for a few hundred steps on LLaVA-Conv (Liu et al., 2023) and ShareGPT4V (Chen et al., 2023), with a large batch size.”
MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI (2023, background): “Meanwhile, large multi-modal models (LMMs) that can understand both text and images have been making a major stride towards more general AI [9, 16, 32, 40, 73].”
Vision-Language Foundation Models as Effective Robot Imitators (2023, methodology): “Motivated by PaLM-E, Brohan et al. (2023) further introduced RT-2, which is based on RT-1 but is adapted to use large vision-language backbones like PaLI-X (Chen et al., 2023) and PaLM-E (Driess et al., 2023), train the policy utilizing robot manipulation data both the web data.”
CogVLM: Visual Expert for Pretrained Language Models (2023, unknown): “Align with the observation from PaLI-X [Chen et al., 2023b], we find it brings no improvement on downstream tasks, although we indeed observed improvements in small models in our early experiments.”
GPT-4V(ision) is a Generalist Web Agent, if Grounded (2024, background): “Many works have focused on improving web agents relying on the HTML document (Deng et al., 2023; Gur et al., 2023; 2022; 2023; Kim et al., 2023; Sridhar et al., 2023).”
OtterHD: A High-Resolution Multi-modality Model (2023, background): “The PaLI series [9, 7, 8] stands out as one of the few research focusing on scaling up the vision encoder.”
VILA: On Pre-training for Visual Language Models (2023, unknown): “There is also usage of different data corpora, including image-text pairs [13, 19, 32, 37], interleaved datasets [6], video-text pairs [41], visual-grounded annotations [36, 45], etc .”
Visual Instruction Tuning with Polite Flamingo (2023, background): “However, there exists a significant challenge yet to be resolved in the process of visual instruction tuning.”