T2VLAD: Global-Local Sequence Alignment for Text-Video Retrieval

By Xiaohan Wang, Linchao Zhu, Yi Yang. 2021

231 citations recorded by the source corpus.

Text-video retrieval is a challenging task that aims to search relevant video contents based on natural language descriptions. The key to this problem is to measure text-video similarities in a joint embedding space. However, most existing methods only consider the global cross-modal similarity and overlook the local details. Some works incorporate the local comparisons through cross-modal local matching and reasoning. These complex operations introduce tremendous computation. In this paper, we design an efficient global-local alignment method. The multi-modal video sequences and text features are adaptively aggregated with a set of shared semantic centers. The local cross-modal similarities are computed between the video feature and text feature within the same center. This design enables the meticulous local comparison and reduces the computational cost of the interaction between each text-video pair. Moreover, a global alignment method is proposed to provide a global cross-modal measurement that is complementary to the local perspective. The global aggregated visual features also provide additional supervision, which is indispensable to the optimization of the learnable semantic centers. We achieve consistent improvements on three standard text-video retrieval benchmarks and outperform the state-of-the-art by a clear margin.

Subfields: Computer Vision and Pattern Recognition, Information Retrieval & Search, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Rongrong Ji
2. Fan Yang
3. Ji Zhang
4. Ming Yan
5. Xiaoshuai Sun
6. Zhongyuan Wang
7. Guohai Xu
8. Shengsheng Qian

In Information Retrieval & Search

1. Rongrong Ji
2. Ming Yan
3. Shengsheng Qian
4. Fan Yang

In Multimodal Learning

1. Rongrong Ji
2. Haoqi Fan
3. Xiaoshuai Sun
4. Ming Yan
5. Shengsheng Qian
6. Guohai Xu
7. Ji Zhang
8. Yiwei Ma

Citations in the corpus

• HiT: Hierarchical Transformer with Momentum Contrast for Video-Text Retrieval (2021, methodology): “Approaches with Dual Streams [14, 40, 13, 62] and HiT, as shown in Figure 4-(c) have become a recent trend for crossmodal retrieval with excellent efficiency.”
• X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrieval (2022, methodology): “Recent years have witnessed the rapid development of VTR, which is supported by a series of pre-training multi-modal models [4, 30, 44], innovative retrieval methods [3, 5, 13–15, 24, 30, 34, 35, 38, 41, 54, 58, 61, 63, 66] and video-text benchmarks [2, 6, 7, 45, 56].”
• Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Loss (2021, background): “T2VLAD (Wang, Zhu, and Yang 2021) extracts features from the aspects of scene and action, and performs similarity matching with the representations of each local token and the global sentence, while HiT (Liu et al. 2021) conducts cross-matching between feature-level and semantic-level embedding.”