Disentangled Representation Learning for Text-Video Retrieval

By Qiang Wang, Yanhao Zhang, Yun Zheng, Pan Pan, Xiansheng Hua. 2022

110 citations recorded by the source corpus.

Cross-modality interaction is a critical component in Text-Video Retrieval (TVR), yet there has been little examination of how different influencing factors for computing interaction affect performance. This paper first studies the interaction paradigm in depth, where we find that its computation can be split into two terms, the interaction contents at different granularity and the matching function to distinguish pairs with the same semantics. We also observe that the single-vector representation and implicit intensive function substantially hinder the optimization. Based on these findings, we propose a disentangled framework to capture a sequential and hierarchical representation. Firstly, considering the natural sequential structure in both text and video inputs, a Weighted Token-wise Interaction (WTI) module is performed to decouple the content and adaptively exploit the pair-wise correlations. This interaction can form a better disentangled manifold for sequential inputs. Secondly, we introduce a Channel DeCorrelation Regularization (CDCR) to minimize the redundancy between the components of the compared vectors, which facilitate learning a hierarchical representation. We demonstrate the effectiveness of the disentangled representation on various benchmarks, e.g., surpassing CLIP4Clip largely by +2.9%, +3.1%, +7.9%, +2.3%, +2.8% and +6.5% R@1 on the MSR-VTT, MSVD, VATEX, LSMDC, AcitivityNet, and DiDeMo, respectively.

Subfields: Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Multimodal Learning

1. Hongwei Xue
2. Jiebo Luo
3. Bei Liu
4. Michael S. Ryoo
5. Houqiang Li
6. Jianlong Fu
7. Andy Zeng
8. Zhe Gan

In Representation & Self-Supervised Learning

1. Andy Zeng
2. Johnny Lee
3. Krzysztof Marcin Choromanski
4. Vikas Sindhwani
5. Michael S. Ryoo
6. Hongwei Xue
7. Houqiang Li
8. Bei Liu

Citations in the corpus

• Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, methodology): “0 is roughly comparable to the R@1 of the best finetuned-SOTA method, DRL (Wang et al. 2022) on the entire 1k-A dataset, with 56.”
• CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Alignment (2022, background): “They transfer CLIP from views of feature aggregation (Luo et al., 2021; Zhao et al., 2022; Fang et al., 2021; Gorti et al., 2022) or representation alignment (Fang et al., 2021; Gorti et al., 2022; Wang et al., 2022c).”