Expanding Language-Image Pretrained Models for General Video Recognition

By Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xiang, Haibin Ling. European Conference on Computer Vision, 2022

522 citations recorded by the source corpus.

Contrastive language-image pretraining has shown great success in learning visual-textual joint representation from web-scale data, demonstrating remarkable"zero-shot"generalization ability for various image tasks. However, how to effectively expand such new language-image pretraining methods to video domains is still an open problem. In this work, we present a simple yet effective approach that adapts the pretrained language-image models to video recognition directly, instead of pretraining a new model from scratch. More concretely, to capture the long-range dependencies of frames along the temporal dimension, we propose a cross-frame attention mechanism that explicitly exchanges information across frames. Such module is lightweight and can be plugged into pretrained language-image models seamlessly. Moreover, we propose a video-specific prompting scheme, which leverages video content information for generating discriminative textual prompts. Extensive experiments demonstrate that our approach is effective and can be generalized to different video recognition scenarios. In particular, under fully-supervised settings, our approach achieves a top-1 accuracy of 87.1% on Kinectics-400, while using 12 times fewer FLOPs compared with Swin-L and ViViT-H. In zero-shot experiments, our approach surpasses the current state-of-the-art methods by +7.6% and +14.9% in terms of top-1 accuracy under two popular protocols. In few-shot scenarios, our approach outperforms previous best methods by +32.1% and +23.1% when the labeled data is extremely limited. Code and models are available at https://aka.ms/X-CLIP

View Paper: https://www.semanticscholar.org/paper/36ee474d55d08465f9df76d290bf5a190d74db65

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey (2024, methodology): “…a range of PEFT strategies have been proposed and implemented in VLA models, such as semantic segmentation [193], [205], [196], point cloud understanding [219], [241], [181], [81], video understanding [89], [138], [114], visual reasoning [63], [41], temporal action detection [137], to name a few.”
A Cookbook of Self-Supervised Learning (2023, methodology): “The frame features can also be used directly, by appending a linear layer on top of the features [Radford et al., 2019], or by using more complex heads [Ni et al., 2022, Arnab et al., 2021].”
Image as a Foreign Language: BEIT Pretraining for Vision and Vision-Language Tasks (2023, methodology): “We directly adopt the framework of X-CLIP [12] for Kinetics-400 and keep all the hyperparameters, except the learning rate, the same for a fair comparison.”
Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models (2023, methodology): “Numerous attempts have also been made to adapt CLIP for video applications [17, 16].”