Align and Prompt: Video-and-Language Pre-training with Entity Prompts

By Dongxu Li, Junnan Li, Hongdong Li, Juan Carlos Niebles, Steven C. H. Hoi. 2021

230 citations recorded by the source corpus.

Yidco-and-language pre-training has shown promising improvements on various downstream tasks. Most previous methods capture cross-modal interactions with a standard transformer-based multimodal encoder, not fully addressing the misalignment between unimodal video and text features. Besides, learning finegrained visual-language alignment usually requires off-the-shelf object detectors to provide object information, which is bottlenecked by the detector's limited vocabulary and expensive computation cost. In this paper, we propose Align and Prompt: a new video-and-language pre-training framework (AlPro), which operates on sparsely-sampled video frames and achieves more effective cross-modal alignment without explicit object detectors. First, we introduce a video-text contrastive (VTC) loss to align unimodal video-text features at the instance level, which eases the modeling of cross-modal interactions. Then, we propose a novel visually-grounded pre-training task, prompting entity modeling (PEM), which learns finegrained alignment between visual region and text entity via an entity prompter module in a self-supervised way. Finally, we pretrain the video-and-language transformer models on large webly-source video-text pairs using the proposed VTC and PEM losses as well as two standard losses of masked language modeling (MLM) and video-text matching (VTM). The resulting pre-trained model achieves state-of-the-art performance on both text-video retrieval and videoQA, outperforming prior work by a substantial margin. Implementation and pre-trained models are available at https://github.com/salesforce/ALPRO.

Subfields: Prompting & Reasoning, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Prompting & Reasoning

In Multimodal Learning

Citations in the corpus

Zero-Shot Video Question Answering via Frozen Bidirectional Language Models (2022, methodology): “However, these works typically use it to learn generic visual-linguistic representations by updating the transformer weights, and then use expensive manual supervision to train randomly initialized task-specific answer classifiers for VQA [15, 25, 57, 58, 65, 69, 90, 91, 96, 99, 106, 118] or VideoQA”
Multimodal Learning With Transformers: A Survey (2022, methodology): “For instance, two-stream architectures [9], [263] are typically preferred over one-stream ones for cross-modal retrievallike tasks in efficiency, since the representation of each modality can be pre-computed beforehand and reused repeatedly.”
Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers (2024, methodology): “The standard protocols [17, 24, 34, 39, 78] jointly use 3M images from CC3M [64] and 2.5M videos as the pretraining datasets.”
Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning (2023, background): “[52] Dongxu Li, Junnan Li, Hongdong Li, Juan Carlos Niebles, and Steven CH Hoi.”
A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities (2022, methodology): “ALPRO [216] follows CLIP’s [205] manual template, and its main contribution is using contrast learning to further align individual modalities in the embedding space.”
HiTeA: Hierarchical Temporal-Aware Video-Language Pre-training (2022, methodology): “Second, some works [2, 11, 28, 32] switch the image encoder to video encoder for learning spatiotemporal contexts within videos.”