Multi-modal Transformer for Video Retrieval

By Valentin Gabeur, Chen Sun, Alahari Karteek, Cordelia Schmid. 2020

723 citations recorded by the source corpus.

The task of retrieving video content relevant to natural language queries plays a critical role in effectively handling internet-scale datasets. Most of the existing methods for this caption-to-video retrieval problem do not fully exploit cross-modal cues present in video. Furthermore, they aggregate per-frame visual features with limited or no temporal information. In this paper, we present a multi-modal transformer to jointly encode the different modalities in video, which allows each of them to attend to the others. The transformer architecture is also leveraged to encode and model the temporal information. On the natural language side, we investigate the best practices to jointly optimize the language embedding together with the multi-modal transformer. This novel framework allows us to establish state-of-the-art results for video retrieval on three datasets. More details are available at http://thoth.inrialpes.fr/research/MMT.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

FLAVA: A Foundational Language And Vision Alignment Model (2021, background): “The self-supervised pretraining paradigm has significantly advanced the state of the art across various domains, from natural language processing [6, 17–19, 23, 24, 28, 30, 61, 68, 73, 83–85], to computer vision [2, 5, 8, 12, 31, 33, 37, 59, 75, 104, 117], to speech recognition [4, 22, 42, 67, 119] ”
VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling (2021, methodology): “To enhance the performance, there have been parallel interests in bringing in more modalities from raw video inputs [31,47,48] and end-to-end training till the raw pixels [18,19,25,49], both aiming to elevate video representations for VidL modeling.”
Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “Methods using such readily-available data have shown significant improvements on several tasks including video retrieval, action localization, action recognition and video captioning [25, 56, 57, 58, 71, 72, 100], sometimes outperforming fully-supervised baselines.”
Less is More: CLIPBERT for Video-and-Language Learning via Sparse Sampling (2021, background): “3[15] shows that adding audio features greatly improves performance.”
Multiscale Vision Transformers (2021, background): “An emerging thread of work aims at applying transformers to vision tasks such as object detection [5], semantic segmentation [115, 99], 3D reconstruction [72], pose estimation [107], generative modeling [14], image retrieval [27], medical image segmentation [13, 97, 111], point clouds [40], video in”
Support-set bottlenecks for video-text representation learning (2020, methodology): “For this, we follow MMT (Gabeur et al., 2020), but with two important diﬀerences.”
VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation (2021, background): “However, existing works [41, 26, 29, 20, 50, 44] in this field are often evaluated on distinct datasets under different experimental settings, making fair comparison difficult between methods.”
VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding (2021, background): “For example, (Miech et al., 2019, 2020; Ging et al., 2020; Gabeur et al., 2020; Alayrac et al., 2020; Patrick et al., 2021; Huang et al., 2021) adopt contrastive learning for pre-training and shows the possibility of zero-shot transfer to text-video retrieval tasks.”
CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval (2021, methodology): “Recently, the pre-trained models (Zhu and Yang, 2020; Amrani et al., 2021; Luo et al., 2020; Li et al., 2020; Miech et al., 2020; Gabeur et al., 2020; Patrick et al., 2021; Lei et al., 2021; Dzabraev et al., 2021; Liu et al., 2021) have dominated the leaderboard of the video-text retrieval with…”
Look Before you Speak: Visually Contextualized Utterances (2020, methodology): “Single-Stream Multimodal Baseline: We also implement a single stream transformer operating on a single multimodal input stream, which is the most widely used framework for video encoding with multimodal inputs [25, 42, 48, 77].”
HiT: Hierarchical Transformer with Momentum Contrast for Video-Text Retrieval (2021, methodology): “We follow MMT [13] to conduct pre-trained feature extraction.”
HANet: Hierarchical Alignment Networks for Video-Text Retrieval (2021, background): “Some works [12, 25, 28] made full use of multimodal cues, e.”
UniFormer: Unifying Convolution and Self-Attention for Visual Recognition (2022, background): “Furthermore, other works propose different variants for spatiotemporal representation learning [1], [3], [5], [27], [57], [64], such as video object tracking [88], video object segmentation [24], video retrieval [25], [30] and video superresolution [7].”
Audio Retrieval with Natural Language Queries (2021, background): “The majority of recent works that address the text-based video retrieval task focuses heavily on the visual and text domains [1, 2, 6, 7].”
Cross-Modality Fusion Transformer for Multispectral Object Detection (2021, methodology): “Multimodal Transformers have been applied to various tasks including image segmentation [24, 25], cross-modal sequence generation[26, 27], video retrieval[28, 29] and image/video captioning/classification [30– 34].”
T2VLAD: Global-Local Sequence Alignment for Text-Video Retrieval (2021, methodology): “Notably, we outperform a HowTo100M-pretrained [25] multi-modal transformer [9] with 2.”
Align and Prompt: Video-and-Language Pre-training with Entity Prompts (2021, methodology): “Most prior methods on text-video retrieval [13,31,42,52,54] and videoQA [11, 14, 24, 26] employ pre-trained visual backbones and extract video features for each frame densely yet offline.”