Cross-Modal and Hierarchical Modeling of Video and Text

By Bowen Zhang, Hexiang Hu, Fei Sha. 2018

208 citations recorded by the source corpus.

Visual data and text data are composed of information at multiple granularities. A video can describe a complex scene that is composed of multiple clips or shots, where each depicts a semantically coherent event or action. Similarly, a paragraph may contain sentences with different topics, which collectively conveys a coherent message or story. In this paper, we investigate the modeling techniques for such hierarchical sequential data where there are correspondences across multiple modalities. Specifically, we introduce hierarchical sequence embedding (hse), a generic model for embedding sequential data of different modalities into hierarchically semantic spaces, with either explicit or implicit correspondence information. We perform empirical studies on large-scale video and paragraph retrieval datasets and demonstrated superior performance by the proposed methods. Furthermore, we examine the effectiveness of our learned embeddings when applied to downstream tasks. We show its utility in zero-shot action recognition and video captioning.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

All in One: Exploring Unified Video-Language Pre-Training (2022, background): “Method Frames R1 R5 R10 MdR FSE [49] 16 13.”
Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval (2021, neutral): “R@1 R@5 R@10 MedR S2VT [64] 11.9 33.6 - 13.0 FSE [77] 13.9 36.0 - 11.0 CE [38] 16 (5% improvement in R@1).”
Less is More: CLIPBERT for Video-and-Language Learning via Sparse Sampling (2021, methodology): “For DiDeMo and ActivityNet Captions, we follow [80, 41] to evaluate paragraph-to-video retrieval, where all sentence descriptions for a video are concatenated”
Multi-modal Transformer for Video Retrieval (2020, methodology): “of the video. In this work, we instead estimate both a video embedding and a caption Multi-modal Transformer for Video Retrieval 5 embedding and then compute the similarity between them. Zhang et al. [33] perform paragraph-to-video retrieval by assuming a hierarchical decomposition of the video and ”
Support-set bottlenecks for video-text representation learning (2020, methodology): “Following Gabeur et al. (2020); Zhang et al. (2018), we concatenate descriptions of segments in a video to construct “video-paragraph” for retrieval and captioning.”
VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding (2021, neutral): “In Table 3, zero-shot VideoCLIP outperforms most supervised DiDeMo dataset R@1 ↑ R@5 S UPERVISED S2VT (Venugopalan et al., 2014) 11.9 33.6 FSE (Zhang et al., 2018) 13.9 44.5 CE (Liu et al., 2019a) 16.1 41.1 ClipBERT (Lei et al., 2021) 20.4 48.0 Z ERO - SHOT VideoCLIP (Zero-shot) 16.6 46.9 Table 2:…”
CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval (2021, methodology): “We follow the setting from (Zhang et al., 2018; Gabeur et al., 2020) to concatenate all the descriptions of a video to form a paragraph and evaluate the model with videoparagraph retrieval on the ‘val1’ split.”
X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrieval (2022, methodology): “Recent years have witnessed the rapid development of VTR, which is supported by a series of pre-training multi-modal models [4, 30, 44], innovative retrieval methods [3, 5, 13–15, 24, 30, 34, 35, 38, 41, 54, 58, 61, 63, 66] and video-text benchmarks [2, 6, 7, 45, 56].”
Fine-Grained Video-Text Retrieval With Hierarchical Graph Reasoning (2020, unknown): “[44].”
HANet: Hierarchical Alignment Networks for Video-Text Retrieval (2021, unknown): “Another two works [24, 50] are also hierarchical models, however, the method in [50] is not applicable to decompose single sentences, and the hierarchical transformer in [24] only focus on global features.”
TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignment (2021, methodology): “Another line of work learns ﬁne-grained or hierarchical alignment be-tween videos and texts [56, 49, 6].”
CLIP2Video: Mastering Video-Text Retrieval via Image CLIP (2021, methodology): “In the past years, remarkable progress [40, 43, 18, 21, 13, 11, 27, 41, 20, 9, 6, 3] has occurred across many video-text benchmarks [15, 39, 38, 7, 32, 16].”
T2VLAD: Global-Local Sequence Alignment for Text-Video Retrieval (2021, background): “The compared baselines include HSE [2], CE [22], HSE [39], and MMT [9].”