COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learning

By Simon Ging, Mohammadreza Zolfaghari, Hamed Pirsiavash, Thomas Brox. 2020

181 citations recorded by the source corpus.

Many real-world video-text tasks involve different levels of granularity, such as frames and words, clip and sentences or videos and paragraphs, each with distinct semantics. In this paper, we propose a Cooperative hierarchical Transformer (COOT) to leverage this hierarchy information and model the interactions between different levels of granularity and different modalities. The method consists of three major components: an attention-aware feature aggregation layer, which leverages the local temporal context (intra-level, e.g., within a clip), a contextual transformer to learn the interactions between low-level and high-level semantics (inter-level, e.g. clip-video, sentence-paragraph), and a cross-modal cycle-consistency loss to connect video and text. The resulting method compares favorably to the state of the art on several benchmarks while having few parameters. All code is available open-source at this https URL

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding (2021, neutral): “Task-specific Alignment Pre-training Coot(Ging et al., 2020) 16.”
Transformers in Vision: A Survey (2021, background): “Long-range dependency modeling is desirable in various uni-modal and multi-modal learning tasks such as activity recognition [24, 68, 98, 173, 214].”
VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding (2021, background): “For example, (Miech et al., 2019, 2020; Ging et al., 2020; Gabeur et al., 2020; Alayrac et al., 2020; Patrick et al., 2021; Huang et al., 2021) adopt contrastive learning for pre-training and shows the possibility of zero-shot transfer to text-video retrieval tasks.”
HANet: Hierarchical Alignment Networks for Video-Text Retrieval (2021, background): “Video-text retrieval is a non-trivial branch of cross-modal retrieval [3, 13, 15, 21, 27, 31, 36, 40, 46].”
TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts (2022, methodology): “More recent methods consider the use of various deep learning apparatus such as GANs [17,31], deep reinforcement learning [10,38], and transformers [8,12].”
HiT: Hierarchical Transformer with Momentum Contrast for Video-Text Retrieval (2021, background): “For Video-Text Retrieval task, we are given M videos V = {Vi}M−1i=0 and N captions T = {Ti} N−1 i=0 .”