Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models

By Muhammad Maaz, Hanoona Abdul Rasheed, Salman Hameed Khan, Fahad Shahbaz Khan. Annual Meeting of the Association for Computational Linguistics, 2023

1,248 citations recorded by the source corpus.

Conversation agents fueled by Large Language Models (LLMs) are providing a new way to interact with visual data. While there have been initial attempts for image-based conversation models, this work addresses the under-explored field of \emph{video-based conversation} by introducing Video-ChatGPT. It is a multimodal model that merges a video-adapted visual encoder with an LLM. The resulting model is capable of understanding and generating detailed conversations about videos. We introduce a new dataset of 100,000 video-instruction pairs used to train Video-ChatGPT acquired via manual and semi-automated pipeline that is easily scalable and robust to label noise. We also develop a quantitative evaluation framework for video-based dialogue models to objectively analyze the strengths and weaknesses of video-based dialogue models. Code: https://github.com/mbzuai-oryx/Video-ChatGPT.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/bf7025a2e5dbb3c09deae02a1aa98a256ca559e2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding (2023, background): “With advancements in Large Language Models (LLMs), numerous studies have endeavored to integrate LLMs with a video encoder, thereby harnessing the powerful comprehension and generation capabilities of LLMs for video tasks [13, 18, 23, 28, 29, 40, 57].”
ShareGPT4Video: Improving Video Understanding and Generation with Better Captions (2024, methodology): “This has inspired a shift towards video understanding[34, 1, 32, 69, 42, 40] and generation tasks[54, 20, 60, 57, 4, 39], allowing for user interaction across video and language modalities.”
mPLUG-OwI2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration (2023, background): “Data sets mPLUG-Owl2 is first pre-trained on image-text”
VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding (2025, neutral): “(2) Open-Ended Video Question Answering (OE-VQA) tasks, including ActivityNet-QA [129] and VCGBench [25].”
A survey on multimodal large language models (2023, neutral): “…LLaVA-Med [37], MiniGPT-4 [13], X-LLM [25], MultiInstruct [26] GPT-aided PMC-VQA [30], VideoChat [33], DetGPT [38], LAMM [39], VisionLLM [24], MultiModal-GPT [31], LLaVA [21], ChatBridge [29], LLaVA-Med [37], GPT4Tools [34], Video-ChatGPT [40], Macaw-LLM [41] Adversarial robustness [45] Figure 2.”
GeoChat:Grounded Large Vision-Language Model for Remote Sensing (2023, background): “More recent studies have shown that these models can be adapted to other domains such as videos [22], biomedical [14, 29] and remote sensing [12].”
Kosmos-2: Grounding Multimodal Large Language Models to the World (2023, neutral): “On SEED-Bench, we compare K OSMOS -2 with popular MLLMs including MiniGPT4 (Zhu et al., 2023), LLaVA (Liu et al., 2023a), BLIP-2 (Li et al., 2023b), InstructBLIP (Dai et al., 2023), MultiModal-GPT (Gong et al., 2023), mPLUG-Owl (Ye et al., 2023), VideoChat (Li et al., 2023c) and Video-ChatGPT (Maaz ”
VideoChat: chat-centric video understanding (2023, neutral): “However, if they just wanted a brief glimpse of the man singinganditsatisfiestheircuriosity,theymaynotfeelcheatedat all.”
PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning (2024, methodology): “1: Video-ChatGPT [30] fails in data scaling.”
Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding (2023, background): “These models are characterized by their effectiveness in capturing and translating complex visual patterns into coherent linguistic representations [5, 12, 18, 33, 45, 49, 71, 73, 78].”
MovieChat: From Dense Token to Sparse Memory for Long Video Understanding (2023, methodology): “Existing vision-centric MLLMs follow the paradigm that utilizing pre-trained LLMs and visual encoder with additional learnable modules (Q-former [21, 34, 36, 82] or simple projection layer [22, 39, 43, 61]).”
Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding (2023, methodology): “Our evaluation protocol follows that of Maaz et al. [40], utilizing GPT-assisted evaluation to assess the capabilities of models.”
InternVideo2: Scaling Foundation Models for Multimodal Video Understanding (2024, methodology): “Some video-centric MLLMs have been proposed, such as VideoChat [Li et al., 2023c], VideoChatGPT [Maaz et al., 2023b] and Valley [Luo et al., 2023], by using instruction data to connect video encoders to LLMs for open-world video understanding.”
MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (2023, methodology): “For tasks associated with action and pose (a)(b)(c)(d)(e)(p) , our VideoChat2 and VideoChat [42] tends to outperform VideoChatGPT [51], underscoring the significance of elaborate video back-bones [41, 43] for effective action and pose recognition.”