Vision-Language Models for Vision Tasks: A Survey

By Jingyi Zhang, Jiaxing Huang, Sheng Jin, Shijian Lu. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023

1,489 citations recorded by the source corpus.

Most visual recognition studies rely heavily on crowd-labelled data in deep neural networks (DNNs) training, and they usually train a DNN for each single visual recognition task, leading to a laborious and time-consuming visual recognition paradigm. To address the two challenges, Vision-Language Models (VLMs) have been intensively investigated recently, which learns rich vision-language correlation from web-scale image-text pairs that are almost infinitely available on the Internet and enables zero-shot predictions on various visual recognition tasks with a single VLM. This paper provides a systematic review of visual language models for various visual recognition tasks, including: (1) the background that introduces the development of visual recognition paradigms; (2) the foundations of VLM that summarize the widely-adopted network architectures, pre-training objectives, and downstream tasks; (3) the widely-adopted datasets in VLM pre-training and evaluations; (4) the review and categorization of existing VLM pre-training methods, VLM transfer learning methods, and VLM knowledge distillation methods; (5) the benchmarking, analysis and discussion of the reviewed methods; (6) several research challenges and potential research directions that could be pursued in the future VLM studies for visual recognition.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/690df0820f35a47e1ce44f90e6ddb4132aa09267

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

RemoteCLIP: A Vision Language Foundation Model for Remote Sensing (2023, background): “The integration of image and text content has been a longstanding challenge and a focal point of research in the field of AI [40], [41], [42], [43].”
R1-VL: Learning to Reason with Multimodal Large Language Models via Step-Wise Group Relative Policy Optimization (2025, neutral): “Multimodal Large Language Models (MLLMs) [1, 7, 13, 16, 19, 34, 40, 46, 47] have shown remarkable advancements across a wide range of vision-language understanding tasks, demonstrating their capabilities in comprehending and analyzing visual contents across various application domains.”
The Segment Anything Model (SAM) for Remote Sensing Applications: From Zero to One Shot (2023, background): “…studied, but one that surged in recent years, with the advancements of Visual Foundation Models (VFM) and Large Multimodal Models (LMM), is known as "Promptable Segmentation," an approach that aims to create a versatile model capable of adapting to a variety of segmentation tasks [39, 71].”
A Survey on Vision--Language--Action Models for Embodied AI (2024, neutral): “[14] examined more recent vision, language, and robotic foundation models for general-purpose robots.”
Foundation Models Defining a New Era in Vision: A Survey and Outlook (2025, neutral): “The work of [28], [29] summarizes different vision-language pre-training network architectures, objectives, and downstream tasks and catego-rizes vision-language pre-training frameworks.”
A Training-Free Approach to Long-Tail Object Detection for Autonomous Driving (2026, unknown): “In parallel, multimodal foundation models, especially vision-language models (VLMs) [21] , have demonstrated remarkable generalization capabilities.”