HiT: Hierarchical Transformer with Momentum Contrast for Video-Text Retrieval

By Song Liu, Haoqi Fan, Shengsheng Qian, Yiru Chen, Wenkui Ding, Zhongyuan Wang. 2021

172 citations recorded by the source corpus.

Video-Text Retrieval has been a hot research topic with the growth of multimedia data on the internet. Transformer for video-text learning has attracted increasing attention due to its promising performance. However, existing cross-modal transformer approaches typically suffer from two major limitations: 1) Exploitation of the transformer architecture where different layers have different feature characteristics is limited; 2) End-to-end training mechanism limits negative sample interactions in a mini-batch. In this paper, we propose a novel approach named Hierarchical Transformer (HiT) for video-text retrieval. HiT performs Hierarchical Cross-modal Contrastive Matching in both feature-level and semantic-level, achieving multi-view and comprehensive retrieval results. Moreover, inspired by MoCo, we propose Momentum Cross-modal Contrast for cross-modal learning to enable large-scale negative sample interactions on-the-fly, which contributes to the generation of more precise and discriminative representations. Experimental results on the three major Video-Text Retrieval benchmark datasets demonstrate the advantages of our method.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Retrieval & Search, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Retrieval & Search

In Multimodal Learning

Citations in the corpus

VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling (2021, background): “To enhance the performance, there have been parallel interests in bringing in more modalities from raw video inputs [31,47,48] and end-to-end training till the raw pixels [18,19,25,49], both aiming to elevate video representations for VidL modeling.”
Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval (2021, background): “MoEE [43], CE [38], MMT [21] and concurrent work HiT [37] all follow this paradigm, with the overall similarity for a video-text pair obtained as a weighted sum of each expert’s similarity with the text.”
CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval (2021, background): “, 2021), HiT (Liu et al., 2021), TT-CE+ (Croitoru et al.”
X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrieval (2022, background): “Traditional videotext retrieval methods tend to design task-specific or modalityspecific fusion strategies for cross-modal learning from offline extracted video and text features [15, 16, 20, 28, 34, 42, 62], including face recognition/object recognition/audio processing.”
HANet: Hierarchical Alignment Networks for Video-Text Retrieval (2021, methodology): “Another two works [24, 50] are also hierarchical models, however, the method in [50] is not applicable to decompose single sentences, and the hierarchical transformer in [24] only focus on global features.”
Video-Text Pre-training with Learned Regions (2021, background): “Video-Text pre-training [6,34,40,41,47,65], which aims to learn transferable representations by aligning the semantics of video and text, has attracted researchers’ attention in recent years.”
Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Loss (2021, background): “T2VLAD (Wang, Zhu, and Yang 2021) extracts features from the aspects of scene and action, and performs similarity matching with the representations of each local token and the global sentence, while HiT (Liu et al. 2021) conducts cross-matching between featurelevel and semantic-level embedding.”
Disentangled Representation Learning for Text-Video Retrieval (2022, methodology): “Over the past few years, most TVR studies have focused on improving performance by increasing the power of text and visual encoders [6,12,25,26].”
CLIP2Video: Mastering Video-Text Retrieval via Image CLIP (2021, background): “In the past years, remarkable progress [40, 43, 18, 21, 13, 11, 27, 41, 20, 9, 6, 3] has occurred across many video-text benchmarks [15, 39, 38, 7, 32, 16].”