Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Loss

By Xingyi Cheng, Hezheng Lin, Xiangyu Wu, Fan Yang, Dong Shen. 2021

187 citations recorded by the source corpus.

Employing large-scale pre-trained model CLIP to conduct video-text retrieval task (VTR) has become a new trend, which exceeds previous VTR methods. Though, due to the heterogeneity of structures and contents between video and text, previous CLIP-based models are prone to overfitting in the training phase, resulting in relatively poor retrieval performance. In this paper, we propose a multi-stream Corpus Alignment network with single gate Mixture-of-Experts (CAMoE) and a novel Dual Softmax Loss (DSL) to solve the two heterogeneity. The CAMoE employs Mixture-of-Experts (MoE) to extract multi-perspective video representations, including action, entity, scene, etc., then align them with the corresponding part of the text. In this stage, we conduct massive explorations towards the feature extraction module and feature alignment module. DSL is proposed to avoid the one-way optimum-match which occurs in previous contrastive methods. Introducing the intrinsic prior of each pair in a batch, DSL serves as a reviser to correct the similarity matrix and achieves the dual optimal match. DSL is easy to implement with only one-line code but improves significantly. The results show that the proposed CAMoE and DSL are of strong efficiency, and each of them is capable of achieving State-of-The-Art (SOTA) individually on various benchmarks such as MSR-VTT, MSVD, and LSMDC. Further, with both of them, the performance is advanced to a big extend, surpassing the previous SOTA methods for around 4.6\% R@1 in MSR-VTT.

Subfields: Computer Vision and Pattern Recognition, Information Retrieval & Search, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Yu Qiao
2. Luc Van Gool
3. Dacheng Tao
4. Andrew Zisserman
5. Xiaogang Wang
6. Ming-Hsuan Yang
7. Jitendra Malik
8. Trevor Darrell

In Information Retrieval & Search

1. Krzysztof Marcin Choromanski
2. Vikas Sindhwani
3. Houqiang Li
4. Jiebo Luo
5. Andy Zeng

In Multimodal Learning

1. Hongwei Xue
2. Jiebo Luo
3. Bei Liu
4. Yu Qiao
5. Yali Wang
6. Yinan He
7. Yi Wang
8. Michael S. Ryoo

Citations in the corpus

• Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, methodology): “Finetuning JEMC (Mithun et al. 2018) 12.5 32.1 42.4 16.0 yes Collaborative Experts (Liu et al. 2019a) 20.6 50.3 64.0 5.3 15.6 40.9 55.2 8.3 yes SSB (Patrick et al. 2020) 28.5 58.6 71.6 3.0 no CLIP4Clip (Luo et al. 2021) 43.1 70.5 81.2 2.0 no ViT-V/32 CLIP2Video (Fang et al. 2021) 43.5 72.3 82.1 2.0 ”
• CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Alignment (2022, methodology): “* denotes using post-processing DSL (Cheng et al., 2021). combined with ImageCaption.”
• InternVideo: General Video Foundation Models via Generative and Discriminative Learning (2022, methodology): “To boost model performance, we also adopt the dual softmax loss [91] as the post-processing operation.”
• CLIP2TV: An Empirical Study on Transformer-based Methods for Video-Text Retrieval (2021, methodology): “Even without it, we acquire a comparable performance to CAMoE [2] in text-to-video retrieval.”