Revisiting the “Video” in Video-Language Understanding

By Shyamal Buch, Cristobal Eyzaguirre, Adrien Gaidon, Jiajun Wu, Li Fei-Fei, Juan Carlos Niebles. 2022

231 citations recorded by the source corpus.

What makes a video task uniquely suited for videos, beyond what can be understood from a single image? Building on recent progress in self-supervised image-language models, we revisit this question in the context of video and language tasks. We propose the atemporal probe (ATP), a new model for video-language analysis which provides a stronger bound on the baseline accuracy of multimodal models constrained by image-level understanding. By applying this model to standard discriminative video and language tasks, such as video question answering and text-to-video retrieval, we characterize the limitations and potential of current video-language benchmarks. We find that understanding of event temporality is often not necessary to achieve strong or state-of-the-art performance, even compared with recent large-scale video-language models and in contexts intended to benchmark deeper video-level understanding. We also demonstrate how ATP can improve both video-language dataset and model design. We describe a technique for leveraging ATP to better disentangle dataset subsets with a higher concentration of temporally challenging data, improving benchmarking efficacy for causal and temporal understanding. Further, we show that effectively integrating ATP into full video-level temporal models can improve efficiency and state-of-the-art accuracy. 11Project website: https://stanfordvl.github.io/atp-revisit-video-lang/

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Pan Zhou
2. Shuicheng Yan
3. Fei Huang
4. Haiyang Xu
5. Tat-seng Chua
6. Ji Zhang
7. Qi Qian
8. Carl Vondrick

In Computer Vision and Pattern Recognition

1. Shuicheng Yan
2. Dacheng Tao
3. Luc Van Gool
4. Tat-seng Chua
5. Pan Zhou
6. Trevor Darrell
7. Yi Yang
8. Bernt Schiele

In Multimodal Learning

1. Shuicheng Yan
2. Fei Huang
3. Tat-seng Chua
4. Qi Qian
5. Haiyang Xu
6. Carl Vondrick
7. Ming Yan
8. Pan Zhou

Citations in the corpus

• ViperGPT: Visual Inference via Python Execution for Reasoning (2023, methodology): “Despite seeing no video data whatsoever, ViperGPT achieves accuracy results on par with the best supervised model (see Table 4), and even surpassing it on the NeXT-QA hard split [7], both for temporal and causal queries.”
• VideoAgent: Long-form Video Understanding with Large Language Model as Agent (2024, unknown): “Importantly, VideoAgent achieves remarkable performance improvements on the more challenging subsets, ATP-hard [4], demonstrating its adeptness at addressing complex long-form video queries.”
• HiTeA: Hierarchical Temporal-Aware Video-Language Pre-training (2022, methodology): “[4] filter out the trivial question for the dataset, and build the hard split for causality and temporal related questions for evaluate the causality and temporal of the model.”
• Video Graph Transformer for Video Question Answering (2022, methodology): “VGT even outperforms a latest work ATP [6] which is based on CLIP features [44] (VGT vs.”