Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond

By Jinze Bai, Shuai Bai, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, Jingren Zhou. 2023

2,258 citations recorded by the source corpus.

In this work, we introduce the Qwen-VL series, a set of large-scale vision-language models (LVLMs) designed to perceive and understand both texts and images. Starting from the Qwen-LM as a foundation, we endow it with visual capacity by the meticulously designed (i) visual receptor, (ii) input-output interface, (iii) 3-stage training pipeline, and (iv) multilingual multimodal cleaned corpus. Beyond the conventional image description and question-answering, we implement the grounding and text-reading ability of Qwen-VLs by aligning image-caption-box tuples. The resulting models, including Qwen-VL and Qwen-VL-Chat, set new records for generalist models under similar model scales on a broad range of visual-centric benchmarks (e.g., image captioning, question answering, visual grounding) and different settings (e.g., zero-shot, few-shot). Moreover, on real-world dialog benchmarks, our instruction-tuned Qwen-VL-Chat also demonstrates superiority compared to existing vision-language chatbots. Code, demo and models are available at https://github.com/QwenLM/Qwen-VL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/fc6a2f7478f68adefd69e2071f27e38aa1647f2f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents (2024, positive): “…and proposed several grounding strategies, including 1) Choice , where the planner is asked to choose from a short list of filtered HTML elements, and 2) SoM , where the input screenshot is superposed with Set-of-Mark (Yang et al., 2023) labels and the planner is asked to select from the labels.”
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone (2024, background): “In this setup, the prompts include instructions to select a single letter corresponding to an answer from a list of given options, or answer with a single word or phrase.”
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook (2024, neutral): “Significant progress has been made in the field of general-domain image-text M-LLMs through the development of models such as GLaMM [71], Qwen-VL [72], SpatialVLM [73], InternVL [74], Osprey [75], Vary [76], ShareGPT4V [77], OtterHD [78], LION [79], SPHINX [80], BLIVA [81], SVIT [82], LLaVA [46],…”
Instruction Tuning for Large Language Models: A Survey (2023, neutral): “For example, MT-Bench [46] includes only 80 test instructions, while AlpacaEval [68] features many straightforward tasks, such as “What is the capital of Australia?””
Otter: A Multi-Modal Model With In-Context Instruction Tuning (2023, positive): “…which follows natural language instructions effectively to accomplish real-world tasks and allows for customizing task-speciﬁc rules into instructions during downstream ﬁne-tuning, enabling pre-trained models to comprehend user intents more effectively and produce accurate and relevant responses.”
Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models (2024, methodology): “We adopt the general architecture used by many recent VLMs, such as LLaVa, Qwen-VL, and PaLI-3 (Liu et al., 2023c; Bai et al., 2023; Chen et al., 2023b).”
MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI (2023, background): “(ix) Qwen-VL [5] introduces a set of trainable query embeddings and single-layer cross-attention module to bridge the modalities, supporting interleaved image-text input.”
What matters when building vision-language models? (2024, positive): “…2023; Sun et al., 2023; Radford et al., 2021) enables researchers to build upon these unimodal pre-trained models to create advanced VLMs that solve these problems with increasing accuracy (Dai et al., 2023; Liu et al., 2023; Bai et al., 2023; Lin et al., 2023; Li et al., 2023; Wang et al., 2023).”
EarthGPT: A Universal Multimodal Large Language Model for Multisensor Image Comprehension in Remote Sensing Domain (2024, background): “5 [6], Sphinx [77], and Qwen-VL-Chat [97] incorrectly categorize the storage tank as a stadium or residential area.”
MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems? (2024, methodology): “…(LLMs) as the text-only baseline, which only take textual questions as input, including ChatGPT [45] and GPT-4 [46], (b) Closed-source MLLMs , represented by models like GPT-4V [47], Gemini-Pro [22], Qwen-VL-Max [3], and Qwen-VL-Plus, and (c) Open-source MLLMs , featuring models such as LLaVA-1.”
LLaVA-OneVision: Easy Visual Task Transfer (2024, positive): “In subsequent collections, we accumulated open-sourced datasets from the Internet and referred to the dataset collection processes of other advanced LMMs, such as Qwen-VL [8], DeepSeek-VL [89], Intern-VL [22], Vision-Flan [145], UReader [149], Idefics-2 (Cauldron) [63], and Cambrian.”
OtterHD: A High-Resolution Multi-modality Model (2023, methodology): “However, the amount of our instruction tuning training is relatively small compared to state-of-the-art LMMs [31, 4], there’s a possibility that Fuyu’s original capabilities might be compromised to some extent.”
BLINK: Multimodal Large Language Models Can See but Not Perceive (2024, background): “…in the modern era of large language models (LLMs), we, as a community, have focused less on such perceptual tasks, and instead have developed new tasks, mostly expressed in natural language, emphasizing the vision-language connection learned by multimodal LLMs [3,6,18,20,24,26,50,51,56,62,71,77].”
SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models (2023, background): “LLaMA-2-70B shows a marked improvement in the few-shot setting over the zero-shot setting, increasing from 2.41% to 8.40%.”
Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action (2023, methodology): “We hypothesize that because it requires external knowledge, extensive language pre-training is important for this task, and therefore our reduced performance is since U NIFIED -IO 2 was not pre-trained as extensively on text as the dedicated language models used by Qwen-VL [12] and mPLUG-Owl2 [206].”
Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs (2024, methodology): “There are three primary approaches to connector design: Resamplers [5], Q-Formers [10, 31], and MLP Projectors [73, 75].”
Qwen Technical Report (2023, methodology): “Note that we previously released the multimodal LLM, QWEN-VL and QWEN-VLCHAT (Bai et al., 2023), which are also based on our QWEN base models.”
Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks (2024, methodology): “This method also includes training large language models on different visual question-answering datasets to unify tasks, like LLaVA [34], Instruct-BLIP [12], Qwen-VL [3] and other MLLMs [60, 40, 80].”
DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models (2024, methodology): “We use Qwen-VL [1] as our default large vision-language model, which exhibits remarkable performance in tasks like question answering, visual localization, and text recognition.”