ShareGPT4V: Improving Large Multi-Modal Models with Better Captions

By Lin Chen, Jinsong Li, Xiao-wen Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, Dahua Lin. European Conference on Computer Vision, 2023

1,159 citations recorded by the source corpus.

In the realm of large multi-modal models (LMMs), efficient modality alignment is crucial yet often constrained by the scarcity of high-quality image-text data. To address this bottleneck, we introduce the ShareGPT4V dataset, a pioneering large-scale resource featuring 1.2 million highly descriptive captions, which surpasses existing datasets in diversity and information content, covering world knowledge, object properties, spatial relationships, and aesthetic evaluations. Specifically, ShareGPT4V originates from a curated 100K high-quality captions collected from advanced GPT4-Vision and has been expanded to 1.2M with a superb caption model trained on this subset. ShareGPT4V first demonstrates its effectiveness for the Supervised Fine-Tuning (SFT) phase, by substituting an equivalent quantity of detailed captions in existing SFT datasets with a subset of our high-quality captions, significantly enhancing the LMMs like LLaVA-7B, LLaVA-1.5-13B, and Qwen-VL-Chat-7B on the MME and MMBench benchmarks, with respective gains of 222.8/22.0/22.3 and 2.7/1.3/1.5. We further incorporate ShareGPT4V data into both the pre-training and SFT phases, obtaining ShareGPT4V-7B, a superior LMM based on a simple architecture that has remarkable performance across a majority of the multi-modal benchmarks. This project is available at https://ShareGPT4V.github.io to serve as a pivotal resource for advancing the LMMs community.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/f68f6f2a057c4e6e5a3c91fc8563533d9bf6e560

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation (2024, neutral): “25 million image-text paired captions from ShareGPT 4 V [10] for multimodal understanding and approximately 1 .”
VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs (2024, methodology): “Additionally, we mix image captioning data from ShareGPT4V (Chen et al., 2023b) and image VQA and instruction-following data from LLaVA (Liu et al., 2023b,a) to maintain the capabilities in understanding static visual concepts.”
LlaVA-CoT: Let Vision Language Models Reason Step-By-Step (2024, neutral): “ShareGPT4V [9] General VQA 31.3k ChartQA [40] General VQA 17.2k A-OKVQA [48] General VQA 16.1k AI2D [25] Science-Targeted VQA 11.4k GeoQA+ [7] Science-Targeted VQA 11.4k ScienceQA [36] Science-Targeted VQA 5.6k DocVQA [41] General VQA 4.0k PISC [30] General VQA 1.0k CLEVR [24] General VQA 0.5k…”
MMBench: Is Your Multi-modal Model an All-around Player? (2023, methodology): “The learning paradigm and the multimodal instruction tuning corpus proposed by LLaVA are widely adopted by subsequent works [32, 8, 2, 11].”
Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models (2025, positive): “Those general benchmarks are used to evaluate the data quality of our proposed Vision-R1-cold dataset.”
Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, positive): “…[203], Laion-EN [203], Laion-COCO [204], LLaVAR [305], InternVL-SA-1B-Caption [113], Captioning MMInstruct [155], GRIT-Caption [194], ShareGPT4V [29], LVIS-Instruct-4V [244], ShareCaptioner [29], OmniCorpus [133], ShareGPT4o [35] GQA [98], OKVQA [178], A-OKVQA [205], Visual7W [317], VisText…”
Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Multimodal Models (2024, methodology): “…training data may either be proprietary ( e.g ., [5]) or, in cases where it is released, there is a heavy reliance on synthetic data generated by proprietary systems, e.g ., models are trained on datasets like ShareGPT4V [7] which uses GPT-4V [25] to generate a large set of detailed image captions.”
Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning (2024, neutral): “However, many popular MLLMs are primarily trained to respond to instructions based on visual inputs, employing a decoder-only autoregressive design as a single black box.”
How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites (2024, positive): “Additionally, the evaluation was mainly supported by VLMEvalKit [23].”
LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (2024, methodology): “Specifically, PLLaVA, VideoChat2, VideoLLaVA and ShareGPT4Video [Xu et al., 2024, Lin et al., 2023, Chen et al., 2023, 2024] are fed with 16 frames, while LLaVA-Next-Video (34B/M7B) [Zhang et al., 2024] are fed with 32 frames.”
Instruction Tuning for Large Language Models: A Survey (2023, neutral): “This section synthesizes key insights gained from these methods and their relative advantages.”
ShareGPT4Video: Improving Video Understanding and Generation with Better Captions (2024, background): “The vision-language community has recently made significant advancements in the field of large image-language models [36, 35, 37, 10, 13, 11, 58, 59, 2, 72, 33].”
SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training (2025, neutral): “…perception include combining multiple visual encoders (Tong et al., 2024d; Kar et al., 2025; Tong et al., 2024a), curating high-quality SFT data (Chen et al., 2023; Liu et al., 2024; Tong et al., 2024a), and improving the SFT training recipe by unfreezing the visual backbone (Liu et al., 2023;…”
VLMEvalKit: An Open-Source ToolKit for Evaluating Large Multi-Modality Models (2024, background): “Most of the research has focused on LMMs for image and text [5,29], but research has also been extended to other modalities, such as audiotext [10], video [7, 25, 40], or point clouds [46, 47].”
What matters when building vision-language models? (2024, positive): “We fine-tune Idefics2 for a few hundred steps on LLaVA-Conv (Liu et al., 2023) and ShareGPT4V (Chen et al., 2023), with a large batch size.”
Are We on the Right Way for Evaluating Large Vision-Language Models? (2024, methodology): “Moreover, TinyLLaVA [53], despite its modest 3B scale, outperformed some competitors of 7B and even 13B surprisingly, underscoring the potential of smaller-scale LVLMs. Additionally, even with the same architecture, ShareGPT4V-7B [5] even outperforms LLaVA-1.”
Show-o: One Single Transformer to Unify Multimodal Understanding and Generation (2024, methodology): “Although there are various design choices of MLLM (McKinzie et al., 2024; Tong et al., 2024), such as vision encoder, feature alignment adapter, and dataset, the training for most of these models adheres to the autoregressive generation paradigm, which is shown to be an effective approach of text-ge”
A survey on multimodal large language models (2023, neutral): “Although existing benchmark datasets can contribute a rich source of data, they usually do not well meet human needs in real-world scenarios, such as multiple rounds of conversations.”
Otter: A Multi-Modal Model With In-Context Instruction Tuning (2023, neutral): “Concurrently, to maintain OpenFlamingo’s in-context learning capacity, we retrieve in-context examples for each triplet, which often lack correlated context, such as a visual question-answer data sample in VQAv2 [3].”