Learning a Text-Video Embedding from Incomplete and Heterogeneous Data

By Antoine Miech, Ivan Laptev, Josef Sivic. 2018

251 citations recorded by the source corpus.

Joint understanding of video and language is an active research area with many applications. Prior work in this domain typically relies on learning text-video embeddings. One difficulty with this approach, however, is the lack of large-scale annotated video-caption datasets for training. To address this issue, we aim at learning text-video embeddings from heterogeneous data sources. To this end, we propose a Mixture-of-Embedding-Experts (MEE) model with ability to handle missing input modalities during training. As a result, our framework can learn improved text-video embeddings simultaneously from image and video datasets. We also show the generalization of MEE to other input modalities such as face descriptors. We evaluate our method on the task of video retrieval and report results for the MPII Movie Description and MSR-VTT datasets. The proposed MEE model demonstrates significant improvements and outperforms previously reported methods on both text-to-video and video-to-text retrieval tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Self-Supervised MultiModal Versatile Networks (2020, background): “With larger datasets [48, 69, 71, 92, 96], many works have profited from learning such a joint visual-textual space [17, 19, 29, 30, 41, 53, 57, 63, 68, 84, 85, 87, 88, 93].”
End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, neutral): “We compare the NCE based approach (ours) to the frequently used max margin ranking loss [19, 21, 33, 40, 51, 55, 79, 80, 83, 84] and a binary classification loss (i.”
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, methodology): “In this work, we use the class of non-linear embedding functions used in [32], which are given by:”
Affine Invariant Analysis of Frank-Wolfe on Strongly Convex Sets (2020, background): “They form a practical family of algorithms [1, 3, 10, 19, 26, 29, 30, 34, 37, 40]; however, many open questions remain in designing such optimal algorithmic schemes (e.”
Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers (2021, background): “Additionally, inspired by the success of (e.g., van den Oord et al., 2018) for self-supervised representation learning, we examine whether using a contrastive image–text matching loss instead of a classification one improves the quality of representations in our models.”
ActBERT: Learning Global-Local Video-Text Representations (2020, background): “, video question answering [36, 10, 18, 54], video captioning [46, 52], textvideo retrieval [47, 41, 25], video grounding [50].”
Contrastive Bidirectional Transformer for Temporal Representation Learning (2019, background): “Language can be leveraged by finding a joint embedding space for both visual and textual modalities or by learning an alignment between the two modalities [14, 2, 38].”
Multi-modal Transformer for Video Retrieval (2020, methodology): “describe those three components. Features F. In order to learn an eective representation from dierent modalities inherent in video data, we begin with video feature extractors called \experts" [14,16,19,31]. In contrast to previous methods, we learn a joint representation leveraging both cross-”
TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval (2020, methodology): “Model references: MCN [1], CAL [8], MEE [22], ExCL [10].”
AVLnet: Learning Audio-Visual Language Representations from Instructional Videos (2020, background): “These models often do not incorporate the audio signal, but even models that do [16–22] still require text captions.”
Noise Estimation Using Density Estimation for Self-Supervised Multimodal Learning (2020, methodology): “[12,97,10,20] for VQA, and [36,46,99,42,43,41] for Text-ToVideo Retrieval.”
Look Before you Speak: Visually Contextualized Utterances (2020, background): “Popular vision and language tasks include visual question answering [4, 5, 22, 28, 58, 61], visual dialog [15, 16, 36, 40, 67], visual captioning [56, 59, 68, 73, 86], visual grounding [18, 22, 57, 83] and video-text retrieval [7, 25, 49, 53, 63].”
Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddings (2019, methodology): “Instead of the combined loss in Equation (7), we use the pair loss, used also in [22]: This same loss is used when we train different MMENs.”
ActionCLIP: A New Paradigm for Video Action Recognition (2021, background): “Vision-text multi-modality is a hot topic in several vision-text related fields recently, like pre-training [20, 21], vision-text retrieval [10, 31] and so on.”
Temporal Localization of Moments in Video Collections with Natural Language (2019, methodology): “We also tried retrieving 200 videos with MEE followed by MCN with TEF for re-ranking and found that our approach outperforms this baseline on all criteria for Charades-STA and for K ∈ {1, 10} on DiDeMo (we are on par for K = 100).”
A Straightforward Framework For Video Retrieval Using CLIP (2021, methodology): “For instance, the state-of-the-art in video retrieval has been pushed by models that implement a Mixture-ofExperts (MoE) paradigm [5,7,10,13].”