Prompting Visual-Language Models for Efficient Video Understanding

By Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, Weidi Xie. 2021

522 citations recorded by the source corpus.

Image-based visual-language (I-VL) pre-training has shown great success for learning joint visual-textual representations from large-scale web data, revealing remarkable ability for zero-shot generalisation. This paper presents a simple but strong baseline to efficiently adapt the pre-trained I-VL model, and exploit its powerful ability for resource-hungry video understanding tasks, with minimal training. Specifically, we propose to optimise a few random vectors, termed as continuous prompt vectors, that convert video-related tasks into the same format as the pre-training objectives. In addition, to bridge the gap between static images and videos, temporal information is encoded with lightweight Transformers stacking on top of frame-wise visual features. Experimentally, we conduct extensive ablation studies to analyse the critical components. On 10 public benchmarks of action recognition, action localisation, and text-video retrieval, across closed-set, few-shot, and zero-shot scenarios, we achieve competitive or state-of-the-art performance to existing methods, despite optimising significantly fewer parameters.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Prompting & Reasoning, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Prompting & Reasoning

In Multimodal Learning

Citations in the corpus

Visual Prompt Tuning (2022, background): “Although prompting has also been applied to visionlanguage models recently [61,85,35,77,18], prompting is still limited to the input of text encoders.”
Conditional Prompt Learning for Vision-Language Models (2022, background): “In computer vision, prompt learning is a nascent research direction that has only been explored very recently [27, 42, 56,58,62].”
Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters (2024, background): “The success of efficient tuning strategies in NLP promotes their applications on vision-language models [22, 34, 65, 76, 79] like CLIP [22].”
ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning for Action Recognition (2022, methodology): “[31] adopted the CLIP model for video recognition tasks by learning videospecific prompts.”
Knowledge Conflicts for LLMs: A Survey (2024, unknown): “LLMs evolve to process information across various formats, including text, images (Alayrac et al., 2022; Li et al., 2023b), video (Ju et al., 2022; Zhang et al., 2023b), and audio (Borsos et al., 2023; Wu et al., 2023).”
See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoning (2023, unknown): “Later, prompting was brought to the vision community [24,31,34,59,74,75].”
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey (2024, methodology): “…a range of PEFT strategies have been proposed and implemented in VLA models, such as semantic segmentation [193], [205], [196], point cloud understanding [219], [241], [181], [81], video understanding [89], [138], [114], visual reasoning [63], [41], temporal action detection [137], to name a few.”
Expanding Language-Image Pretrained Models for General Video Recognition (2022, methodology): “One is ActionCLIP [49], while the other is [21].”