CogVLM: Visual Expert for Pretrained Language Models

By Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song and 6 more. Neural Information Processing Systems, 2023

827 citations recorded by the source corpus.

We introduce CogVLM, a powerful open-source visual language foundation model. Different from the popular shallow alignment method which maps image features into the input space of language model, CogVLM bridges the gap between the frozen pretrained language model and image encoder by a trainable visual expert module in the attention and FFN layers. As a result, CogVLM enables deep fusion of vision language features without sacrificing any performance on NLP tasks. CogVLM-17B achieves state-of-the-art performance on 10 classic cross-modal benchmarks, including NoCaps, Flicker30k captioning, RefCOCO, RefCOCO+, RefCOCOg, Visual7W, GQA, ScienceQA, VizWiz VQA and TDIUC, and ranks the 2nd on VQAv2, OKVQA, TextVQA, COCO captioning, etc., surpassing or matching PaLI-X 55B. Codes and checkpoints are available at https://github.com/THUDM/CogVLM.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/2313afae52d98e569da2dedbf14daf9efc74e7cf

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

VILA: On Pre-training for Visual Language Models (2023, neutral): “Most of the VLM pre-training [33, 37, 58] relies on image-text pairs ( i.e ., image and captions) due to the wide availability and large diversity ( e.g ., LAION [49], COYO [10]).”
Are We on the Right Way for Evaluating Large Vision-Language Models? (2024, background): “5 series [24], ShareGPT4V-7B [5], Monkey-Chat [23], LLaVA-Next [25], Deepseek-VL-7B [28], LLaVA-Next-34B [25], CogVLM-Chat-17B [45], InternVL-Chat-v1.”
What matters when building vision-language models? (2024, positive): “…2023; Sun et al., 2023; Radford et al., 2021) enables researchers to build upon these unimodal pre-trained models to create advanced VLMs that solve these problems with increasing accuracy (Dai et al., 2023; Liu et al., 2023; Bai et al., 2023; Lin et al., 2023; Li et al., 2023; Wang et al., 2023).”
ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment (2024, methodology): “To generate long, highly descriptive captions, we apply the state-of-the-art MLLM CogVLM [53] as auto-captioner to synthesize image descriptions with the main objects and surroundings while specifying the corresponding color, texture, spatial relationships, etc .”
ShareGPT4Video: Improving Video Understanding and Generation with Better Captions (2024, background): “Recent advancements in multi-modal learning, driven by large language models, have led to progress in image-text dialogue[35, 10, 16, 56, 63, 13, 2, 65] and text-to-image generation tasks[3, 52, 6, 53, 70, 8].”
InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models (2025, neutral): “Multimodal large language models (MLLMs) [32, 66, 121, 21, 19, 123, 68, 114, 97, 136, 71, 31, 85, 117, 18, 89, 105, 69] have recently achieved or even surpassed human-level performance in a broad spectrum of tasks, underscoring their potential as a significant stride toward artificial general…”
CogAgent: A Visual Language Model for GUI Agents (2023, positive): “As our base VLM, We select CogVLM-17B [38], an open-sourced and state-of-the-art large vison-language model.”
SpatialRGPT: Grounded Spatial Reasoning in Vision Language Model (2024, background): “KOSMOS-2 [21], Shikra [22], MiniGPT-2 [23], CogVLM [24], SPHINX [25], and LLaVA [26] have enabled MLLMs to achieve region-based image understanding.”
Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models (2024, background): “As multimodal language models (LMs) [35, 42, 28, 27, 1, 3, 49, 7, 31, 41, 6, 5] have begun to mature, we now expect them to solve tasks like the ones mentioned above, i.e., ones where people draw intermediate sketches to simplify reasoning.”
BLINK: Multimodal Large Language Models Can See but Not Perceive (2024, methodology): “( viii) CogVLM [77] adds a trainable visual expert module in the attention and FFN layers to bridge different modalities better.”
GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest (2023, positive): “We use the feature map of the penultimate transformer layer as the representation of the entire image and then map it to the language space using a single linear layer.”
VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs (2024, methodology): “…2022; Bai et al., 2023), Q-Former (Li et al., 2023a; Zhu et al., 2023b), Linear Projection (Liu et al., 2023b; Chen et al., 2023a) and Dynamic Visual Tokenizer (Jin et al., 2023c), have been widely adopted in popular Video-LLMs (Zhang et al., 2023; Maaz et al., 2023b; Lin et al., 2023a; Li et al.,…”
A survey on multimodal large language models (2023, neutral): “Then we structure the remaining content as illustrated in Fig.”
VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks (2024, neutral): “Wang et al. (2023b) introduced trainable visual expert modules to improve the fusion between the vision and language models.”
Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, neutral): “In recent years, multimodal large language models (MLLMs) [60, 137, 246, 36, 35, 248, 140, 228, 192, 275, 143, 54, 170] have emerged as a pivotal technology in artificial intelligence, capable of processing and understanding information from multiple modalities such as text, images, and videos.”
LVBench: An Extreme Long Video Understanding Benchmark (2024, neutral): “Building upon the achievements in LLMs, the field has shifted towards MLLMs to enhance multi-modal understanding capabilities [2, 11, 26, 32, 38].”