VideoAgent: Long-form Video Understanding with Large Language Model as Agent

By Xiaohan Wang, Yuhui Zhang, Orr Zohar, S. Yeung-Levy. European Conference on Computer Vision, 2024

359 citations recorded by the source corpus.

Long-form video understanding represents a significant challenge within computer vision, demanding a model capable of reasoning over long multi-modal sequences. Motivated by the human cognitive process for long-form video understanding, we emphasize interactive reasoning and planning over the ability to process lengthy visual inputs. We introduce a novel agent-based system, VideoAgent, that employs a large language model as a central agent to iteratively identify and compile crucial information to answer a question, with vision-language foundation models serving as tools to translate and retrieve visual information. Evaluated on the challenging EgoSchema and NExT-QA benchmarks, VideoAgent achieves 54.1% and 71.3% zero-shot accuracy with only 8.4 and 8.2 frames used on average. These results demonstrate superior effectiveness and efficiency of our method over the current state-of-the-art methods, highlighting the potential of agent-based approaches in advancing long-form video understanding.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/567157a4722d79f5ac019142e5fb1ad10b4e86cb

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Yifei Xin
2. Sicong Leng
3. Wenqi Zhang
4. Li Bing
5. Guanzheng Chen
6. Hang Zhang
7. Deli Zhao
8. Yongxin Zhu

Citations in the corpus

• MLVU: A Comprehensive Benchmark for Multi-Task Long Video Understanding (2024, neutral): “Additionally, it is also explored to make selective usage of frames or clips from long videos based on retrievers or agents [46; 33; 40].”
• VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs (2024, background): “…problems, like long video understanding (Ren et al., 2023; Song et al., 2023; Wang et al., 2024c), video agent (Lin et al., 2023b; Fan et al., 2024; Wang et al., 2024a; He et al., 2024), autonomous driving (Xu et al., 2023; Shao et al., 2024), motion understanding (Wu et al., 2024; Chen et al.,…”