DeepSeek-VL: Towards Real-World Vision-Language Understanding

By Haoyu Lu, Wen Liu, Bo Zhang, Bing-Li Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang and 5 more. arXiv.org, 2024

847 citations recorded by the source corpus.

We present DeepSeek-VL, an open-source Vision-Language (VL) Model designed for real-world vision and language understanding applications. Our approach is structured around three key dimensions: We strive to ensure our data is diverse, scalable, and extensively covers real-world scenarios including web screenshots, PDFs, OCR, charts, and knowledge-based content, aiming for a comprehensive representation of practical contexts. Further, we create a use case taxonomy from real user scenarios and construct an instruction tuning dataset accordingly. The fine-tuning with this dataset substantially improves the model's user experience in practical applications. Considering efficiency and the demands of most real-world scenarios, DeepSeek-VL incorporates a hybrid vision encoder that efficiently processes high-resolution images (1024 x 1024), while maintaining a relatively low computational overhead. This design choice ensures the model's ability to capture critical semantic and detailed information across various visual tasks. We posit that a proficient Vision-Language Model should, foremost, possess strong language abilities. To ensure the preservation of LLM capabilities during pretraining, we investigate an effective VL pretraining strategy by integrating LLM training from the beginning and carefully managing the competitive dynamics observed between vision and language modalities. The DeepSeek-VL family (both 1.3B and 7B models) showcases superior user experiences as a vision-language chatbot in real-world applications, achieving state-of-the-art or competitive performance across a wide range of visual-language benchmarks at the same model size while maintaining robust performance on language-centric benchmarks. We have made both 1.3B and 7B models publicly accessible to foster innovations based on this foundation model.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/b14e5138d4d1f3577b2390541dc7b730a41bb651

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models (2024, methodology): “DeepSeek-VL (Lu et al., 2024) employs multiple visual encoders to obtain richer visual representations.”
What matters when building vision-language models? (2024, positive): “This results in a high number of visual tokens for each image ranging from 576 for DeepSeek-VL (Lu et al., 2024) to 2890 for SPHINX-2k (Lin et al., 2023).”
Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, positive): “As shown in Figure 2 and Table 2, InternVL 2.5 retains the same model architecture as its predecessors, InternVL 1.5 [35] and InternVL 2.0, following the “ViT-MLP-LLM” paradigm widely adopted in various MLLM studies [150, 151, 36, 316, 162, 246, 124, 256].”
Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation (2024, neutral): “By leveraging pretrained LLMs, MLLMs [1, 2, 12, 49, 53, 79, 92] demonstrate a robust ability to understand and process multimodal information.”
LLaVA-OneVision: Easy Visual Task Transfer (2024, positive): “In subsequent collections, we accumulated open-sourced datasets from the Internet and referred to the dataset collection processes of other advanced LMMs, such as Qwen-VL [8], DeepSeek-VL [89], Intern-VL [22], Vision-Flan [145], UReader [149], Idefics-2 (Cauldron) [63], and Cambrian.”
Are We on the Right Way for Evaluating Large Vision-Language Models? (2024, background): “5 series [24], ShareGPT4V-7B [5], Monkey-Chat [23], LLaVA-Next [25], Deepseek-VL-7B [28], LLaVA-Next-34B [25], CogVLM-Chat-17B [45], InternVL-Chat-v1.”
Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs (2024, background): “Recent work [80, 117] advocates for incorporating these diverse vision models into MLLMs.”
How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites (2024, neutral): “In contrast, open-source models generally train with fixed resolutions [20, 25, 58, 67, 112, 137], such as 336 × 336 and 448 × 448, leading to a considerable gap in capabilities relative to commercial counterparts.”
Towards Text-Image Interleaved Retrieval (2025, unknown): “Second, while fine-tuning multimodal LLMs (MLLMs) with interleaved inputs support (Lu et al., 2024) as dense TIIR models seems promising, the hundreds of visual tokens required per image (Yin et al., 2024) leads to prohibitively long sequences, resulting in both computational inefficiency and…”