Vision-Language Pre-training: Basics, Recent Advances, and Future Trends

By Zhe Gan, Linjie Li, Lijuan Wang, Zicheng Liu, Jianfeng Gao. 2022

223 citations recorded by the source corpus.

This paper surveys vision-language pre-training (VLP) methods for multimodal intelligence that have been developed in the last few years. We group these approaches into three categories: ($i$) VLP for image-text tasks, such as image captioning, image-text retrieval, visual question answering, and visual grounding; ($ii$) VLP for core computer vision tasks, such as (open-set) image classification, object detection, and segmentation; and ($iii$) VLP for video-text tasks, such as video captioning, video-text retrieval, and video question answering. For each category, we present a comprehensive review of state-of-the-art methods, and discuss the progress that has been made and challenges still being faced, using specific systems and models as case studies. In addition, for each category, we discuss advanced topics being actively explored in the research community, such as big foundation models, unified modeling, in-context few-shot learning, knowledge, robustness, and computer vision in the wild, to name a few.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Visual Instruction Tuning (2023, background): “To this end, the community has witnessed an emergent interest in developing language-augmented foundation vision models [27, 16], with strong capabilities in open-world visual understanding such as classification [40, 21, 57, 54, 39], detection [29, 62, 33], segmentation [25, 63, 58] and captioning…”
PaLM-E: An Embodied Multimodal Language Model (2023, background): “, 2020) models, recent years have seen a growing interest in large vision-language models (VLMs) (Li et al., 2019; Lu et al., 2019; Hao et al., 2022; Gan et al., 2022).”
LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day (2023, background): “…inputs, the resulting large multimodal models (LMMs) exhibit strong zero-shot task completion performance on a variety of user-oriented vision-language tasks such as image understanding and reasoning, paving the way to develop general-purpose multimodal conversational assistants [22, 2, 23, 10].”
SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities (2024, background): “Vision language models (VLMs) have made significant progress in recent years across a variety of tasks including image cap-tioning, visual question answering (VQA), embodied planning, action recognition, and more [2, 18, 24, 31].”
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control (2023, background): “Put the strawberry into the correct bowl Pick the nearly falling bag Pick object that is di ﬀ erent Figure 1: RT-2 overview: we represent robot actions as another language, which can be cast into text tokens and trained together with Internet-scale vision-language datasets.”
MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities (2023, unknown): “Vision-language models [13, 31, 51, 14, 44, 36, 72, 69, 74, 25] approach multimodal intelligence of jointly understanding and generating vision and language signals.”
Visual-Language Prompt Tuning with Knowledge-Guided Context Optimization (2023, methodology): “ing [4] [10] [19] [22] [28] [30] [33] [38] has been proposed to adapt the pretrained VLM to downstream tasks, achieving a fantastic performance on various few-shot or zero-shot visual recognization tasks.”
CTP: Towards Vision-Language Continual Pretraining via Compatible Momentum Contrast and Topology Preservation (2023, background): “In contrast, VLP allows for learning open-world visual patterns without explicit “class” concepts, which can capture more comprehensive visual concepts rather than just category-based features.”
Generative Multimodal Models are In-Context Learners (2023, background): “Multimodal tasks [25, 41] encompass anything involving understanding and generation in single or multiple modalities [5, 19, 58], which can be highly diverse and long-tail.”
OCRBench: on the hidden mystery of OCR in large multimodal models (2023, unknown): “The success of large models has also been extended to the multimodal vision-language space [7], leading to a line of research on large multimodal models (LMM), including contrastive learning [8, 9, 10, 11] and generative modeling [2, 12, 13, 14, 15].”