VideoChat: chat-centric video understanding

By Kunchang Li, Yinan He, Yi Wang, Yizhuo Li, Wen Wang, Ping Luo, Yali Wang, Limin Wang, Yu Qiao. Science China Information Sciences, 2023

975 citations recorded by the source corpus.

In this study, we initiate an exploration into video understanding by introducing VideoChat, an end-to-end chat-centric video understanding system. It integrates video foundation models and large language models via a learnable neural interface, excelling in spatiotemporal reasoning, event localization, and causal relationship inference. To instructively tune this system, we propose a video-centric instruction dataset, composed of thousands of videos matched with detailed descriptions and conversations. This dataset emphasizes spatiotemporal reasoning and causal relationships, providing a valuable asset for training chat-centric video understanding systems. Preliminary qualitative experiments reveal our system’s potential across a broad spectrum of video applications and set the standard for future research. Access our code and data at https://github.com/OpenGVLab/Ask-Anything.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/d48cb91b9e555194f7494c4d4bb9815021d3ee45

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought (2023, background): “VideoChat[38], mPLUG-Owl [39] and X-LLM [40], further expand support for video input.”
TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding (2023, methodology): “However, previous work [13, 18, 23, 57] usually set a fixed number of video tokens N V , such as 32, which can result in severe visual semantics degradation when the number of input frames T is large.”
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook (2024, neutral): “Similarly, models such as Video-Chat [47] and Video-LLaMA [48] are designed for video comprehension, whereas SpeechGPT [49] is tailored for audio processing.”
LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models (2024, background): “Traditionally, researchers have approached these challenges by training separate task-specific models for each application scenario, e.g., multi-image [1, 19, 27], video [7, 29, 66], and 3D [14, 15, 58].”
Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers (2024, methodology): “The ﬁve base models include Video-LLaMA [88] (video VQA), VideoChat [38] (video VQA), VideoChat Text [38] (natural language model which textu-alizes the video content), BLIP-2 [37] (image captioning), and MiniGPT-4 [94] (image VQA).”
LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models (2023, methodology): “To support video understanding in LLMs, Video-LLaMA [62] and VideoChat [30] attempt to utilize BLIP-2 for video embedding extraction, while Video-ChatGPT [39] proposes spatial and temporal pooling for video features.”
MovieChat: From Dense Token to Sparse Memory for Long Video Understanding (2023, background): “In video field, some previous works [43,82] follow this paradigm to build video MLLMs, while works in the other paradigm [37, 67] combine existing visual perception tools ( e.g ., tracking and classification) and LLMs through Application Programming In-1 a r X i v : 2307 .”
How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites (2024, neutral): “The development of open-source MLLMs [4, 15, 20, 43, 47, 49, 52, 65, 66, 98, 105, 113, 115, 119, 133, 134] has significantly influenced the AGI landscape by integrating and enhancing capabilities in processing both visual and textual data.”
LaVie: High-Quality Video Generation with Cascaded Latent Diffusion Models (2023, neutral): “These pairs are automatically generated using Videochat (Li et al., 2023).”
VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs (2024, background): “The video-text sources include Panda-70M (Chen et al., 2024b), VIDAL-10M (Zhu et al., 2023a) WebVid-10M (Bain et al., 2021), and InternVid-10M (Wang et al., 2023), while the image-text sources include CC-3M (Changpinyo et al., 2021) and DCI (Urbanek et al., 2023).”
VBench: Comprehensive Benchmark Suite for Video Generative Models (2023, methodology): “VideoChat [64] uses a pre-trained video encoder and a large language model to align and train video VLM with instruction data.”
Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks (2023, background): “Recent advancements have seen the creation of vision large language models (VLLMs) [3, 21, 56, 59, 62, 66, 100, 121, 128, 130, 136, 138, 141, 142, 148], which aim to enhance language models with the capability to process and interpret visual information.”
GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest (2023, neutral): “We also compare the design philosophy with concurrent works that utilize textual coordinates or extra position tokens as region references, all of which can be found in the Appendix A.4.”
A Survey of Hallucination in Large Foundation Models (2023, methodology): “Ideally, the video should enhance the audio, and in (Li et al., 2023a), they have used an advanced language model for data augmentation without human labeling.”
VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding (2025, neutral): “Building upon the success of LLMs, Multimodal LLMs (MLLMs) [7–10] have been proposed for multimodal understanding.”
A survey on multimodal large language models (2023, positive): “For example, VideoChat-Text [33] uses pre-trained vision models to extract visual information such as actions and enriches the descriptions using a speech recognition model.”