TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding

By Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, Lu Hou. Computer Vision and Pattern Recognition, 2023

489 citations recorded by the source corpus.

This work proposes TimeChat, a time-sensitive multi-modal large language model specifically designed for long video understanding. Our model incorporates two key architectural contributions: (1) a timestamp-aware frame encoder that binds visual content with the timestamp of each frame, and (2) a sliding video Q-Former that produces a video token sequence of varying lengths to accommodate videos of various durations. Additionally, we construct an instruction-tuning dataset, encompassing 6 tasks and a total of 125K instances, to further enhance TimeChat's instruction-following performance. Experiment results across various video understanding tasks, such as dense captioning, temporal grounding, and highlight detection, demonstrate TimeChat's strong zero-shot temporal localization and reasoning capabilities. For example, it achieves +9.2 F1 score and +2.8 CIDEr on YouCook2, +5.8 HIT@1 on QVHighlights, and +27.5 R@1 (I oU=0.5) on Charades-STA, compared to state-of-the-art video large language models, holding the potential to serve as a versatile video assistant for long-form video comprehension tasks and satisfy realistic user requirements.11Our code and dataset are available at https://github.com/RenShuhuai-Andy/TimeChat.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/eca8a3e6383e3618e0bc984382e08c09be3cca6c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Yifei Xin
2. Xu Sun
3. Shuhuai Ren
4. Sicong Leng
5. Shicheng Li
6. Wenqi Zhang
7. Peng Jin
8. Li Bing

Citations in the corpus

• MLVU: A Comprehensive Benchmark for Multi-Task Long Video Understanding (2024, unknown): “This category includes: MiniGPT4-Video [3], LLaMA-VID [22], Movie-LLM [36], MA-LMM [12], MovieChat [35], TimeChat [34], and GPT-4o [32].”
• VideoAgent: Long-form Video Understanding with Large Language Model as Agent (2024, unknown): “Meanwhile, selective-compressive methodologies attempt to sub-sample the video into more meaningful frames, utilizing the input question/text as a guide, and in effect attempt to only sample frames relevant to the question at hand [7, 20, 37, 56, 66].”
• Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (2024, unknown): “To empower video MLLMs with temporal localization capability [20, 50, 60], TimeChat [53] constructs time-sensitive instruction tuning datasets and encodes timestamp knowledge into visual tokens.”
• Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding (2023, methodology): “In this work, we introduce Chat-UniVi, designed to not only comprehend and generate responses from text but also incorporate visual inputs, thereby providing a more comprehensive and immersive context for response generation.”
• LVBench: An Extreme Long Video Understanding Benchmark (2024, unknown): “The annotation of long videos presents unique challenges compared to short-form content, requiring careful consideration of temporal dynamics and comprehensive coverage.”
• VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs (2024, background): “…model, VideoLLaMA 2 can be further developed to benefit various more specialized but challenging problems, like long video understanding (Ren et al., 2023; Song et al., 2023; Wang et al., 2024c), video agent (Lin et al., 2023b; Fan et al., 2024; Wang et al., 2024a; He et al., 2024),…”
• LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding (2024, unknown): “LLaMA-VID (Li et al., 2023d) employs an additional text decoder to embed the text query for cross-attention between frame features and compress the context token to one token per frame, while MovieChat (Song et al., 2023) and TimeChat (Ren et al., 2023b) develop memory modules and timestamp-aware en”
• Long Context Transfer from Language to Vision (2024, unknown): “Our proposed context transfer VideoChatGPT-100K [50] 123.4 068.0 LLaVA-Hound-255K [88] 052.4 037.6 ShareGPT4Video[12] 026.6 273.3 TimeIT [60] 190.8 052.5 VideoChat [39] 009.5 059.0 VideoChatGPT [50] 0108.0 NexTQA [77] 0042.9 EgoSchema [51] 0179.8 VideoMME [21] 1017.0 V-NIAH (Ours) 000.”
• TempCompass: Do Video LLMs Really Understand Videos? (2024, background): “Inspired by the unprecedented progress of large language models (LLMs), a line * Equal contribution. of initial efforts (Li et al., 2023c; Zhang et al., 2023; Maaz et al., 2023; Luo et al., 2023; Ren et al., 2023b) have been devoted to build LLMs with video understanding ability.”