MovieChat: From Dense Token to Sparse Memory for Long Video Understanding

By Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Xun Guo, Tianbo Ye, Yang Lu, Jenq-Neng Hwang and 1 more. Computer Vision and Pattern Recognition, 2023

630 citations recorded by the source corpus.

Recently, integrating video foundation models and large language models to build a video understanding system can overcome the limitations of specific pre-defined vision tasks. Yet, existing systems can only handle videos with very few frames. For long videos, the computation complexity, memory cost, and long-term temporal connection impose additional challenges. Taking advantage of the Atkinson-Shiffrin memory model, with tokens in Transformers being employed as the carriers of memory in combination with our specially designed memory mechanism, we propose the MovieChat to overcome these challenges. MovieChat achieves state-of-the-art performance in long video understanding, along with the released MovieChat-1K benchmark with 1K long video and 14K manual annotations for validation of the effectiveness of our method. The code, models and data can be found in https://reself.github.io/MovieChat.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/6f9b7c8cde1be2e62a503c31cac883c6d44c9d0d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Xiao-wen Dong
2. Jiaqi Wang
3. Pan Zhang
4. Xu Sun
5. Haodong Duan
6. Haoning Wu
7. Yuhang Zang
8. Shuhuai Ren

Citations in the corpus

• LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (2024, background): “Existing LMM benchmarks for long videos typically focus on a specific category of videos, e.g. egocentric videos [Mangalam et al., 2023], or movies [Song et al., 2023, Zhang et al., 2023a].”
• VLMEvalKit: An Open-Source ToolKit for Evaluating Large Multi-Modality Models (2024, background): “Most of the research has focused on LMMs for image and text [5,29], but research has also been extended to other modalities, such as audiotext [10], video [7, 25, 40], or point clouds [46, 47].”
• VideoChat: chat-centric video understanding (2023, neutral): “Usually a person watching this video will feel cheated, do you know why?”
• VideoAgent: Long-form Video Understanding with Large Language Model as Agent (2024, neutral): “Compressive sparsity methods [10,12,14,30,42,44,50,53,59,65], attempt to compress the video into meaningful embeddings/representations with the minimum possible dimensionality.”
• PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning (2024, background): “MovieChat [32] implemented a novel memory-based mechanism within transformers, strategically combining similar frames to reduce both computational load and memory footprint.”
• Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook (2024, neutral): “…demonstrated by models such as Video-Chat [47], Video-ChatGPT [107], Video-LLaMA [48], LLaMA-VID [108], MotionGPT [109], LAVENDER [110], MovieChat [111], Vid2Seq [112], VideoLLM [113], and VTimeLLM [114], shows significant promise for the development of models tailored to medical applications.”
• TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding (2023, background): “With advancements in Large Language Models (LLMs), numerous studies have endeavored to integrate LLMs with a video encoder, thereby harnessing the powerful comprehension and generation capabilities of LLMs for video tasks [13, 18, 23, 28, 29, 40, 57].”
• LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding (2024, neutral): “MovieChat (Song et al., 2024) employs a similarity-based frame-level feature selection using visual representation from CLIP (Radford et al., 2021).”
• MLVU: A Comprehensive Benchmark for Multi-Task Long Video Understanding (2024, neutral): “This collection includes 168 movies sourced from the Movie101 dataset [51] and the MovieChat dataset [35], as well as 60 documentaries from the MovieChat dataset [35].”
• LVBench: An Extreme Long Video Understanding Benchmark (2024, neutral): “…[40] 6,678 473 Manual MovieChat-1K [34] 13,000 564 Manual MoVQA [49] 21,953 992 Manual Video-MME [9] 2,700 1018 Manual LVBench(Ours) 1,549 4,101 Manual efficient method for adapting image-language pre-trained models to dense video understanding through a novel feature pooling strategy.”
• Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (2024, neutral): “Regarding MLLMs for processing videos [26, 44, 54], the key difference lies in how they encode the video into vision tokens compatible with the LLMs. Representative work like Video-LLaMA [71] first uses a ViT [14] with an image Q-Former to encode individual frames and then employs a video Q-Former…”
• CFVBench: A Comprehensive Video Benchmark for Fine-grained Multimodal Retrieval-Augmented Generation (2025, unknown): “…modality pairings (e.g., text-video or audio-video), overlooking datasets that simultaneously encompass audio-text-video streams and densely embedded domain-specific formats, such as data charts, complex tables, or dynamic on-screen text, which are critical for real-life video comprehension.”