Fine-Grained Video-Text Retrieval With Hierarchical Graph Reasoning

By Shizhe Chen, Yida Zhao, Qin Jin, Qi Wu. Computer Vision and Pattern Recognition, 2020

381 citations recorded by the source corpus.

Cross-modal retrieval between videos and texts has attracted growing attentions due to the rapid emergence of videos on the web. The current dominant approach is to learn a joint embedding space to measure cross-modal similarities. However, simple embeddings are insufficient to represent complicated visual and textual details, such as scenes, objects, actions and their compositions. To improve fine-grained video-text retrieval, we propose a Hierarchical Graph Reasoning (HGR) model, which decomposes video-text matching into global-to-local levels. The model disentangles text into a hierarchical semantic graph including three levels of events, actions, entities, and generates hierarchical textual embeddings via attention-based graph reasoning. Different levels of texts can guide the learning of diverse and hierarchical video representations for cross-modal matching to capture both global and local details. Experimental results on three video-text datasets demonstrate the advantages of our model. Such hierarchical decomposition also enables better generalization across datasets and improves the ability to distinguish fine-grained semantic differences. Code will be released at \url{https://github.com/cshizhe/hgr_v2t}.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/0b78e14dfc2050878e8c817e4782c0c81ee7f5dd

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models (2021, methodology): “As can be observed, our model with MMP generalizes well on YouTube2Text, outperforming HGR (Chen et al., 2020a) by 7.6 and DualEncoder (Dong et al., 2019) by 10.3 in R@1.”
Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoning (2021, methodology): “Additionally, we also introduce a recent state-of-the-art video-text retrieval model HGR (Chen et al., 2020) for performance comparison, which decomposes video-text matching into global-to-local levels and performs cross-modal matching with attention-based graph reasoning.”
TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignment (2021, background): “Another line of work learns ﬁne-grained or hierarchical alignment be-tween videos and texts [56, 49, 6].”
T2VLAD: Global-Local Sequence Alignment for Text-Video Retrieval (2021, methodology): “The compared baselines include HSE [2], CE [22], HSE [39], and MMT [9].”
CLIP2Video: Mastering Video-Text Retrieval via Image CLIP (2021, methodology): “Since it is inaccessible to obtain test annotation, we report the results following HGR’s [8] validation protocol, which includes 1500 videos for validation and 1500 videos for test.”
COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learning (2020, methodology): “Representation learning based on both vision and language has many potential benefits: visual grounding[1, 2, 3, 4]; visual learning with a more natural, almost self-supervised annotation process; and direct applicability to cross-modal tasks, such as video retrieval by text[5, 6, 7, 8, 9], video su”
Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Loss (2021, background): “Recently, HGR (Chen et al. 2020) is proposed to divide the sentence into three parts: events, action, and entity, with the hypothesis that all the captions own a relatively ﬁxed hierarchical graph form.”
HiT: Hierarchical Transformer with Momentum Contrast for Video-Text Retrieval (2021, methodology): “For Video-Text Retrieval task, we are given M videos V = {Vi}M−1i=0 and N captions T = {Ti} N−1 i=0 .”
Support-set bottlenecks for video-text representation learning (2020, background): “4% , exceeding recent state-of-the-art methods (Chen et al., 2020b) by a margin of 9% .”
Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformers (2021, methodology): “We use the retrieval setup and splits from [6].”
HANet: Hierarchical Alignment Networks for Video-Text Retrieval (2021, positive): “Since there are multiple components in the video and text at the individual level, following [4, 6, 20], we use the stack attention mechanism to align multiple components and compute the overall similarity score.”