ShareGPT4Video: Improving Video Understanding and Generation with Better Captions

By Lin Chen, Xilin Wei, Jinsong Li, Xiao-wen Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang and 5 more. Neural Information Processing Systems, 2024

451 citations recorded by the source corpus.

We present the ShareGPT4Video series, aiming to facilitate the video understanding of large video-language models (LVLMs) and the video generation of text-to-video models (T2VMs) via dense and precise captions. The series comprises: 1) ShareGPT4Video, 40K GPT4V annotated dense captions of videos with various lengths and sources, developed through carefully designed data filtering and annotating strategy. 2) ShareCaptioner-Video, an efficient and capable captioning model for arbitrary videos, with 4.8M high-quality aesthetic videos annotated by it. 3) ShareGPT4Video-8B, a simple yet superb LVLM that reached SOTA performance on three advancing video benchmarks. To achieve this, taking aside the non-scalable costly human annotators, we find using GPT4V to caption video with a naive multi-frame or frame-concatenation input strategy leads to less detailed and sometimes temporal-confused results. We argue the challenge of designing a high-quality video captioning strategy lies in three aspects: 1) Inter-frame precise temporal change understanding. 2) Intra-frame detailed content description. 3) Frame-number scalability for arbitrary-length videos. To this end, we meticulously designed a differential video captioning strategy, which is stable, scalable, and efficient for generating captions for videos with arbitrary resolution, aspect ratios, and length. Based on it, we construct ShareGPT4Video, which contains 40K high-quality videos spanning a wide range of categories, and the resulting captions encompass rich world knowledge, object attributes, camera movements, and crucially, detailed and precise temporal descriptions of events. Based on ShareGPT4Video, we further develop ShareCaptioner-Video, a superior captioner capable of efficiently generating high-quality captions for arbitrary videos...

View Paper: https://www.semanticscholar.org/paper/9583cadea300f67aaab0fdf7b6d1f774c3cd55e7

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Kunchang Li
2. Yi Wang
3. Limin Wang
4. Yu Qiao
5. Yali Wang
6. Zhijian Liu
7. Hongxu Yin
8. Yukang Chen

Citations in the corpus

• Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, unknown): “…PMC-CaseReport [260], GMAI-VL (subset) [134] GUI Screen2Words [240], WebSight [122] Type: Video Datasets Captioning Mementos [254], ShareGPT4Video [30], VideoGPT+ [174], ShareGPT4o-Video [35] General QA VideoChat2-IT [131], EgoTaskQA [99], NTU RGB+D [152], CLEVRER [276], STAR [259], LSMDC [201]…”
• Are We on the Right Way for Evaluating Large Vision-Language Models? (2024, unknown): “Built upon the CLIP image encoder which is somewhat aligned with the language modality, current LVLMs typically utilize vast image-text pairs to connect the vision encoder and LLM, enabling LLM to receive and understand visual content [54, 26, 24, 11, 52, 2, 48, 31, 5].”
• LongVILA: Scaling Long-Context Visual Language Models for Long Videos (2024, methodology): “We evaluate the impact of our two-stage sharding strategy using a video captioning dataset (Chen et al., 2024a).”
• mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models (2024, methodology): “…interleaved format. we include VideoChat2 (Li et al., 2023c), Video-LLaMA2 (Cheng et al., 2024), Video-ChatGPT (Maaz et al., 2023), ShareGPT4Video (Chen et al., 2024c), PLLaVA (Xu et al., 2024), Idefics2 (Laurenccon et al., 2024), Mantis-SigLIP (Jiang et al., 2024) and LLAVA-Interleave (Li et…”
• Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning (2024, unknown): “However, many popular MLLMs are primarily trained to respond to instructions based on visual inputs, employing a decoder-only autoregressive design as a single black box.”
• VLMEvalKit: An Open-Source ToolKit for Evaluating Large Multi-Modality Models (2024, background): “Most of the research has focused on LMMs for image and text [5,29], but research has also been extended to other modalities, such as audiotext [10], video [7, 25, 40], or point clouds [46, 47].”
• Video Instruction Tuning With Synthetic Data (2024, unknown): “For instance, ShareGPT4Video (Chen et al., 2024a) has an average sampling rate of 0.15, sometimes sampling only 2 frames from a 30-second video.”
• VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding (2025, unknown): “While most approaches utilize image-based encoders [14–18, 27 More recent works [22, 23, 150–157] extend beyond visual inputs by incorporating audio, using separate encoders for each modality and integrating them through an LLM decoder.”
• InternVideo2: Scaling Foundation Models for Multimodal Video Understanding (2024, methodology): “In the additional HD training stage, we incorporate the videos with corresponding GPT-4 annotations from [Chen et al., 2024a] into the training set.”
• HunyuanVideo: A Systematic Framework For Large Video Generative Models (2024, unknown): “Most previous work focus on providing either brief captions [14, 50] or dense captions [93, 9, 10].”
• LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding (2024, methodology): “The video training data contains a subset of VideoChat2-IT (Li et al., 2024b), which includes TextVR (Wu et al., 2025), Youcook2 (Zhou et al., 2018), Kinetics-710 (Kay et al., 2017), NExTQA (Xiao et al., 2021), CLEVRER (Yi et al., 2019), EgoQA (Fan, 2019), TGIF (Li et al., 2016), WebVidQA (Yang et a”
• LLaVA-OneVision: Easy Visual Task Transfer (2024, unknown): “For video data, we first integrate the multi-frame data from M4-Instruct, including NExT-QA [141] and ShareGPT4Video [21].”
• Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (2024, unknown): “Representative open-source video MLLMs including Video-LLaVA [33], VideoChat2-Mistral[27], ST-LLM [38], ShareGPT4Video [8],Chat-UniVi-V1.”
• LLaVA-Video: Video Instruction Tuning With Synthetic Data (2024, unknown): “ShareGPT4Video picks key frames using CLIP (Radford et al., 2021) based on frame uniqueness.”