ActionCLIP: A New Paradigm for Video Action Recognition

By Mengmeng Wang, Jiazheng Xing, Yong Liu. 2021

522 citations recorded by the source corpus.

The canonical approach to video action recognition dictates a neural model to do a classic and standard 1-of-N majority vote task. They are trained to predict a fixed set of predefined categories, limiting their transferable ability on new datasets with unseen concepts. In this paper, we provide a new perspective on action recognition by attaching importance to the semantic information of label texts rather than simply mapping them into numbers. Specifically, we model this task as a video-text matching problem within a multimodal learning framework, which strengthens the video representation with more semantic language supervision and enables our model to do zero-shot action recognition without any further labeled data or parameters requirements. Moreover, to handle the deficiency of label texts and make use of tremendous web data, we propose a new paradigm based on this multimodal learning framework for action recognition, which we dub"pre-train, prompt and fine-tune". This paradigm first learns powerful representations from pre-training on a large amount of web image-text or video-text data. Then it makes the action recognition task to act more like pre-training problems via prompt engineering. Finally, it end-to-end fine-tunes on target datasets to obtain strong performance. We give an instantiation of the new paradigm, ActionCLIP, which not only has superior and flexible zero-shot/few-shot transfer ability but also reaches a top performance on general action recognition task, achieving 83.8% top-1 accuracy on Kinetics-400 with a ViT-B/16 as the backbone. Code is available at https://github.com/sallymmx/ActionCLIP.git

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Human-Computer Interaction

Citations in the corpus

All in One: Exploring Unified Video-Language Pre-Training (2022, background): “In recent years, there has been tremendous progress in the development of videolanguage pre-training (VLP) models [11, 21, 38, 50], where joint representations are generally produced with a multimodal fusion Transformer network after extracting the visual and language features through unimodal encod”
Expanding Language-Image Pretrained Models for General Video Recognition (2022, methodology): “We construct 16 hand-craft templates [49] totally.”
Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey (2023, methodology): “Many computer vision tasks are belong to this downstream task, such as pedestrian attribute recognition [248], action recognition [134].”
A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities (2022, methodology): “ActionCLIP [215] is the application of CLIP [205] in action recognition scenarios.”
Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models (2023, background): “Numerous attempts have also been made to adapt CLIP for video applications [17, 16].”
PointCLIP: Point Cloud Understanding by CLIP (2021, background): “CoOp [69], CLIP-Adapter [16], Tip-Adapter [66] and so on [54, 57, 70] further indicate that the performance of CLIP can be largely improved by infusing domain-specific supervisions.”
Prompting Visual-Language Models for Efficient Video Understanding (2021, background): “In the video domains, [51, 73] propose to end-to-end finetune CLIP on individual video tasks, e.”
CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Alignment (2022, background): “…et al., 2021; Jia et al., 2021) have shown powerful capability for various downstream tasks, including visual understanding (Gu et al., 2021; Wang et al., 2021a; Rao et al., 2022) , image-text generation (Patashnik et al., 2021; Mokady et al., 2021) and so on (Guzhov et al., 2022; Zhang et…”
ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning for Action Recognition (2022, unknown): “[74] equipped the CLIP with temporal modules and good performance can be achieved after the model is fully fine-tuned on video datasets.”
InternVideo: General Video Foundation Models via Generative and Discriminative Learning (2022, methodology): “We evaluate our model on Kinetics-400 dataset with 64.25% accuracy, which outperforms the previous SOTA 56.4% [102] by a large margin.”
DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting (2021, methodology): “Thanks to the language supervision, models pre-trained via CLIP achieve impressive results on various visual classification tasks with no or very limited annotations [13, 41, 60].”
Unsupervised Prompt Learning for Vision-Language Models (2022, background): “…to a series of vision tasks, such as object detection (Gu et al. 2021; Du et al. 2022), semantic segmentation (Xu et al. 2021b), action recognition (Wang, Xing, and Liu 2021), video clip retrieval (Luo et al. 2021), video caption (Tang et al. 2021) and 3D recognition (Zhang et al. 2021b However,…”