Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval

By Max Bain, Arsha Nagrani, Gul Varol, Andrew Zisserman. 2021

1,620 citations recorded by the source corpus.

Our objective in this work is video-text retrieval – in particular a joint embedding that enables efficient text-to-video retrieval. The challenges in this area include the design of the visual architecture and the nature of the training data, in that the available large scale video-text training datasets, such as HowTo100M, are noisy and hence competitive performance is achieved only at scale through large amounts of compute.We address both these challenges in this paper. We propose an end-to-end trainable model that is designed to take advantage of both large-scale image and video captioning datasets. Our model is an adaptation and extension of the recent ViT and Timesformer architectures, and consists of attention in both space and time. The model is flexible and can be trained on both image and video text datasets, either independently or in conjunction. It is trained with a curriculum learning schedule that begins by treating images as ‘frozen’ snapshots of video, and then gradually learns to attend to increasing temporal context when trained on video datasets. We also provide a new video-text pretraining dataset WebVid-2M, comprised of over two million videos with weak captions scraped from the internet. Despite training on datasets that are an order of magnitude smaller, we show that this approach yields state-of-the-art results on standard downstream video-retrieval benchmarks including MSR-VTT, MSVD, DiDeMo and LSMDC.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, methodology): “Other methods have also attempted zero-shot on MSR-VTT text-to-video retrieval (Xu et al. 2021; Miech et al. 2020; Bain et al. 2021), but these have all been outperformed by Portillo-Quintero et al. (2021).”
Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “Another family of vision-language models is based on contrastive learning [2, 5, 49, 50, 57, 74, 82, 85, 138, 140, 146].”
Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks (2021, methodology): “The most widely used objectives are image-text retrieval [2, 5, 43, 55, 63, 73, 74, 75], masked language modeling with image clues [2, 23, 43, 55, 72, 73, 74, 75], and masked region modeling [15, 55, 72, 73, 75].”
VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling (2021, methodology): “Impact of different temporal modeling methods over video inputs under different settings: (i) random initialized visual encoder; (ii) ImageNet [84] pre-trained visual encoder and (iii) Adding video-text pre-training on WebVid [25].”
Florence: A New Foundation Model for Computer Vision (2021, neutral): “, 2018), WebVid-2M (Bain et al., 2021), WIT400M (Radford et al.”
VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation (2021, background): “Recent work [31, 44, 9] show that with image-text pre-training, models trained on 2D features alone can achieve decent performance on many video-and-language (VidL) tasks.”
Object-aware Video-language Pre-training for Retrieval (2021, methodology): “The previous works in video-language pretraining focus on aligning the raw-pixel video and raw text with a contrastive loss in both dual-encoder [4,19] and one stream [42] frameworks.”
InternVideo: General Video Foundation Models via Generative and Discriminative Learning (2022, methodology): “The UnlabeledHybrid dataset is used for masked video pretraining, which is consist of Kinetics-710 [56], Something-Something V2 [68], AVA [67], WebVid2M [55], and our self-collected videos.”
Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey (2023, methodology): “44 Frozen [92] ICCV-2021 video/image-text Trans MML Jointly optimize the model on both images and videos 180.”
Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation (2023, positive): “More precisely, we train a UNet for only the 0 to 900 timesteps (with 1000 being the maximum) instead of the typical full range of 0 to 1000, directing the model to be an expert emphasizing high-resolution nuances.”
HiTeA: Hierarchical Temporal-Aware Video-Language Pre-training (2022, methodology): “SSv2-Label SSv2-Template Method # PT Data R@1 R@5 R@10 R@1 R@5 R@10 Frozen [ [48].”
Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding (2023, background): “Visual information serves to augment and enhance our comprehension of the world [5, 23–26, 29, 59].”
ModelScope Text-to-Video Technical Report (2023, background): “WebVid [2].”
Egocentric Video-Language Pretraining (2022, methodology): “The introduction of large-scale video-text datasets [10, 3] has enabled the emergence of VLP approaches to improve the video-text representation for various visionlanguage tasks [21, 22, 4], such as MIL-NCE which [23] proposes to match clips with multiple captions close in temporal to adapt the vide”
Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Loss (2021, positive): “To further prove the generalization of Dual Softmax, we also test it on other methods such as CLIP, FROZEN, and CLIP4Clip.”
Make-A-Video: Text-to-Video Generation without Text-Video Data (2022, methodology): “After all, an image is a video with a single frame (Bain et al., 2021).”
WorldSimBench: Towards Video Generation Models as World Simulators (2024, background): “With the advancement of diffusion transformer (Peebles & Xie, 2023) and the extensive utilization of large-scale internet video datasets (Bain et al., 2021; Ebert et al., 2021; Goyal et al., 2017; Grauman et al., 2022), certain Predictive Actionable Video Model (Yang et al., 2023) models, also…”