VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks

By Wen Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao and 1 more. 2023

699 citations recorded by the source corpus.

Large language models (LLMs) have notably accelerated progress towards artificial general intelligence (AGI), with their impressive zero-shot capacity for user-tailored tasks, endowing them with immense potential across a range of applications. However, in the field of computer vision, despite the availability of numerous powerful vision foundation models (VFMs), they are still restricted to tasks in a pre-defined form, struggling to match the open-ended task capabilities of LLMs. In this work, we present an LLM-based framework for vision-centric tasks, termed VisionLLM. This framework provides a unified perspective for vision and language tasks by treating images as a foreign language and aligning vision-centric tasks with language tasks that can be flexibly defined and managed using language instructions. An LLM-based decoder can then make appropriate predictions based on these instructions for open-ended tasks. Extensive experiments show that the proposed VisionLLM can achieve different levels of task customization through language instructions, from fine-grained object-level to coarse-grained task-level customization, all with good results. It's noteworthy that, with a generalist LLM-based framework, our model can achieve over 60\% mAP on COCO, on par with detection-specific models. We hope this model can set a new baseline for generalist vision and language models. The demo shall be released based on https://github.com/OpenGVLab/InternGPT. The code shall be released at https://github.com/OpenGVLab/VisionLLM.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Multimodal Learning

Citations in the corpus

Improved Baselines with Visual Instruction Tuning (2023, background): “The LLaVA architecture is also lever-aged in different downstream tasks and domains, including region-level [8, 55] and pixel-level [25, 49] understanding, biomedical assistants [30], image generation [5], adversarial studies [6, 58].”
Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation (2024, neutral): “By leveraging pretrained LLMs, MLLMs [1, 2, 12, 49, 53, 79, 92] demonstrate a robust ability to understand and process multimodal information.”
MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models (2023, background): “Speciﬁcally, the ﬁrst manner [49, 18, 44] evaluates on existing traditional multimodal datasets, such as image caption [17] and VQA [23, 39, 35].”
MoE-LLaVA: Mixture of Experts for Large Vision-Language Models (2024, background): “…LVLMs with fine-grained image understanding capabilities, such as region understanding (Chen et al., 2023c; Zhao et al., 2023b; Liu et al., 2023e), multi-region understanding (Wang et al., 2023c; Pi et al., 2023; Peng et al., 2023), and pixel-wise grounding (Rasheed et al., 2023; Lai et al., 2023).”
VTimeLLM: Empower LLM to Grasp Video Moments (2023, background): “Recent endeavors, such as Kosmos-2 [21] and VisionLLM [26], delved into more detailed aspects of image comprehension, including referring and grounding, significantly enhancing the capability to describe intricate image details.”
A survey on multimodal large language models (2023, neutral): “The other works manually design some seed instructions and use these instructions to prompt GPT to generate more [24,31,33].”
Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning (2024, background): “MLLMs have also been extended to various applications, including fine-grained localization [61, 24] such as object detection [72], video understanding [71, 29, 8], and image generation [20, 49].”
PointLLM: Empowering Large Language Models to Understand Point Clouds (2023, background): “They aim at having the capability to interpret diverse modalities, including but not limited to images [13, 18, 27, 45, 56], audio [17], motion [21], etc.”
Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (2024, neutral): “In parallel, the field of computer vision has been striving to develop large autoregressive or world models [42, 68, 5], aiming to emulate their impressive scalability and generalizability.”
SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models (2024, neutral): “For example, Shikra (Chen et al., 2023a) and VisionLLM (Wang et al., 2023b) excel in re-ferring object detection, while ChartAssistant (Meng et al., 2024), mPLUG-DocOwl/PaperOwl (Ye et al., 2023a; Hu et al., 2023) focus on specialized domains such as tables, documents, and scientific diagrams…”
Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action (2023, background): “This includes creating models that can do OCR [12, 220], visual grounding [12, 26, 143, 189, 207, 212, 219], image-text-retrieval [97], additional languages [112], embodied AI tasks [17, 135, 140, 152] or leverage other expert systems [52].”
GPT-Driver: Learning to Drive with GPT (2023, methodology): “We note that language modeling has been employed in other tasks of computer vision and robotics, such as object detection Chen et al. (2021); Xue et al. (2022); Wang et al. (2023) and robotic control Brohan et al. (2023).”