VTimeLLM: Empower LLM to Grasp Video Moments

By Bin Huang, Xin Wang, Hong Chen, Zihan Song, Wenwu Zhu. Computer Vision and Pattern Recognition, 2023

335 citations recorded by the source corpus.

Large language models (LLMs) have shown remarkable text understanding capabilities, which have been ex-tended as Video LLMs to handle video data for compre-hending visual details. However, existing Video LLMs can only provide a coarse description of the entire video, failing to capture the precise start and end time bound-ary of specific events. In this paper, we solve this issue via proposing VTimeLLM, a novel Video LLM designed for fine-grained video moment understanding and reasoning with respect to time boundary. Specifically, our VTimeLLM adopts a boundary-aware three-stage training strategy, which respectively utilizes image-text pairs for feature alignment, multiple-event videos to increase temporal-boundary awareness, and high-quality video-instruction tuning to further improve temporal understanding ability as well as align with human intents. Extensive experiments demonstrate that in fine-grained time-related comprehension tasks for videos such as Temporal Video Grounding and Dense Video Captioning, VTimeLLM significantly outperforms existing Video LLMs. Besides, benefits from the fine-grained temporal understanding of the videos further enable VTimeLLM to beat existing Video LLMs in video di-alogue benchmark, showing its superior cross-modal understanding and reasoning abilities. 11Our project page is at https://github.com/huangb23/VTimeLLM

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/17ff6a0844afe74796022e7aaf372553e9303d72

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sicong Leng
2. Li Bing
3. Guanzheng Chen
4. Hang Zhang
5. Ziyang Luo
6. Xin Li

In Computer Vision and Pattern Recognition

1. Yifei Xin
2. Sicong Leng
3. Wenqi Zhang
4. Li Bing
5. Guanzheng Chen
6. Hang Zhang
7. Deli Zhao
8. Yongxin Zhu

Citations in the corpus

• Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook (2024, neutral): “…demonstrated by models such as Video-Chat [47], Video-ChatGPT [107], Video-LLaMA [48], LLaMA-VID [108], MotionGPT [109], LAVENDER [110], MovieChat [111], Vid2Seq [112], VideoLLM [113], and VTimeLLM [114], shows significant promise for the development of models tailored to medical applications.”
• Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (2024, neutral): “Our extensive evaluation under-scores the need for further advancements in handling longer multimodal data and we hope Video-MME will inspire future research and development in improving the capabilities of MLLMs.”
• VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs (2024, methodology): “…VideoLLaMA 2 can be further developed to benefit various more specialized but challenging problems, like long video understanding (Ren et al., 2023; Song et al., 2023; Wang et al., 2024c), video agent (Lin et al., 2023b; Fan et al., 2024; Wang et al., 2024a; He et al., 2024), autonomous driving…”