Learning Joint Embedding with Multimodal Cues for Cross-Modal Video-Text Retrieval

By Niluthpol Chowdhury Mithun, Juncheng Billy Li, Florian Metze, Amit K. Roy-Chowdhury. 2018

286 citations recorded by the source corpus.

Constructing a joint representation invariant across different modalities (e.g., video, language) is of significant importance in many multimedia applications. While there are a number of recent successes in developing effective image-text retrieval methods by learning joint representations, the video-text retrieval task, however, has not been explored to its fullest extent. In this paper, we study how to effectively utilize available multimodal cues from videos for the cross-modal video-text retrieval task. Based on our analysis, we propose a novel framework that simultaneously utilizes multi-modal features (different visual characteristics, audio inputs, and text) by a fusion strategy for efficient retrieval. Furthermore, we explore several loss functions in training the embedding and propose a modified pairwise ranking loss for the task. Experiments on MSVD and MSR-VTT datasets demonstrate that our method achieves significant performance gain compared to the state-of-the-art approaches.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Self-Supervised MultiModal Versatile Networks (2020, background): “With larger datasets [48, 69, 71, 92, 96], many works have profited from learning such a joint visual-textual space [17, 19, 29, 30, 41, 53, 57, 63, 68, 84, 85, 87, 88, 93].”
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, background): “A common approach to model vision and language is learning a joint embedding space where visual and textual cues are adjacent if and only if they are semantically similar [6, 8, 10, 11, 25, 32, 35, 37, 38, 59, 54, 55, 57].”
Webly Supervised Joint Embedding for Cross-Modal Image-Text Retrieval (2018, background): “Visual-Semantic Embedding: Joint visual-semantic models have shown excellent performance on several multimedia tasks, e.g., cross-modal retrieval [18, 31, 38, 55], image captioning [23, 36], image classification [10, 12, 20] video summarization [4, 41].”
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, methodology): “Finetuning JEMC (Mithun et al. 2018) 12.5 32.1 42.4 16.0 yes Collaborative Experts (Liu et al. 2019a) 20.6 50.3 64.0 5.3 15.6 40.9 55.2 8.3 yes SSB (Patrick et al. 2020) 28.5 58.6 71.6 3.0 no CLIP4Clip (Luo et al. 2021) 43.1 70.5 81.2 2.0 no ViT-V/32 CLIP2Video (Fang et al. 2021) 43.5 72.3 82.1 2.0…”
Multi-modal Transformer for Video Retrieval (2020, methodology): “In order to learn an effective representation from different modalities inherent in video data, we begin with video feature extractors called “experts” [14,16,19,31].”
AVLnet: Learning Audio-Visual Language Representations from Instructional Videos (2020, neutral): “More related to our task is text-video modelling focused on learning a joint multimodal embedding space [3, 4, 31–36].”
Dual Encoding for Zero-Example Video Retrieval (2018, methodology): “Different from [24,34], we propose dual multi-level encoding for both videos and queries in advance to common space learning.”
Noise Estimation Using Density Estimation for Self-Supervised Multimodal Learning (2020, methodology): “…artificial intelligence tasks such as Visual Question Answering (Antol et al. 2015; Jang et al. 2017; Gao et al. 2018; Xu et al. 2017; Fan et al. 2019) and Text-to-Video Retrieval (Liu et al. 2019; Mithun et al. 2018; Yu, Kim, and Kim 2018; Miech, Laptev, and Sivic 2018; Song and Soleymani 2019).”
Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddings (2019, methodology): “Originally used for video cap-tioning, this large-scale video understanding dataset is increasingly evaluated for video-to-text and text-to-video retrieval [8, 22, 24, 41, 23].”
T2VLAD: Global-Local Sequence Alignment for Text-Video Retrieval (2021, methodology): “[26] leveraged a simple text-image embedding method [7] to improve the training strategy with hard negative mining, and incorporated multi-modal features (RBG, motion, and audio) to enrich the video representations.”
COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learning (2020, methodology): “Modalities such as audio, text and motion provide strong cues to learn richer spatio-temporal features [67, 68, 16, 17, 9, 69, 11].”
LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment (2023, background): “These models can capture video semantics and solve downstream tasks (Karpathy et al., 2014; Mithun et al., 2018).”