Jointly Modeling Deep Video and Compositional Text to Bridge Vision and Language in a Unified Framework

By Ran Xu, Caiming Xiong, Jason J. Corso. 2015

325 citations recorded by the source corpus.

Recently, joint video-language modeling has been attracting more and more attention. However, most existing approaches focus on exploring the language model upon on a fixed visual model. In this paper, we propose a unified framework that jointly models video and the corresponding text sentences. The framework consists of three parts: a compositional semantics language model, a deep video model and a joint embedding model. In our language model, we propose a dependency-tree structure model that embeds sentence into a continuous vector space, which preserves visually grounded meanings and word order. In the visual model, we leverage deep neural networks to capture essential semantic information from videos. In the joint embedding model, we minimize the distance of the outputs of the deep video model and compositional language model in the joint space, and update these two models jointly. Based on these three parts, our system is able to accomplish three tasks: 1) natural language generation, and 2) video retrieval and 3) language retrieval. In the experiments, the results show our approach outperforms SVM, CRF and CCA baselines in predicting Subject-Verb-Object triplet and natural sentence generation, and is better than CCA in video retrieval and language retrieval tasks.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Self-Supervised MultiModal Versatile Networks (2020, background): “With larger datasets [48, 69, 71, 92, 96], many works have profited from learning such a joint visual-textual space [17, 19, 29, 30, 41, 53, 57, 63, 68, 84, 85, 87, 88, 93].”
End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, background): “Numerous prior works [19, 21, 27, 28, 42, 51, 55, 60, 62, 88, 79, 80, 83, 84] have used image / video description datasets [48, 63, 65, 87, 90] to learn an embed-”
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, background): “A common approach to model vision and language is learning a joint embedding space where visual and textual cues are adjacent if and only if they are semantically similar [6, 8, 10, 11, 25, 32, 35, 37, 38, 59, 54, 55, 57].”
A Joint Sequence Fusion Model for Video Question Answering and Retrieval (2018, background): “Several deep video-language embedding methods [8,12,13] has been developed by extending image-language embeddings [20,21].”
Movie Description (2016, methodology): “Recently image description has gained increased attention with work such as that of Chen and Zitnick (2015); Donahue et al. (2015); Fang et al. (2015); Karpathy and Fei-Fei (2015); Kiros et al. (2014, 2015); Mao et al. (2015); Vinyals et al. (2015); Xu et al. (2015a).”
Translating Videos to Natural Language Using Deep Recurrent Neural Networks (2014, methodology): “For our task we pick 1200 videos to be used as training data, 100 videos for validation and 670 videos for testing, as used by the prior works on video description (Guadarrama et al., 2013; Thomason et al., 2014; Xu et al., 2015).”
Localizing Moments in Video with Natural Language (2017, methodology): “Existing methods for natural language based video retrieval [21, 48, 43] retrieve an entire video given a text string but do not identify when a moment occurs within a video.”
Learning to Answer Questions from Image Using Convolutional Neural Network (2015, background): “Speciﬁcally, there have been rapid progresses on the task of bidirectional image and sentence retrieval (Socher et al., 2014), and automatic image captioning (Karpathy et al., 2014b; Kiros et al., 2014b; Mao et al., 2014b; Vinyals et al., 2014; Xu et al., 2015).”
Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks (2015, methodology): “To eliminate the tedious effort of rule engineering when the problem scales, some recent methods train statistical models for lexical entries, either in a fully [10, 14, 24, 41] or weakly [36, 35, 57, 54] supervised fashion.”
Jointly Modeling Embedding and Translation to Bridge Video and Language (2015, methodology): “We conduct our experiments mainly on the Microsoft Research Video Description Corpus (YouTube2Text) [5], which have been used in several prior works [11, 29, 34] on action recognition and video description generation tasks.”
Learning a Text-Video Embedding from Incomplete and Heterogeneous Data (2018, background): “One way to leverage language in vision is to find a joint embedding space for both visual and textual modalities [7–9, 19–23].”
Support-set bottlenecks for video-text representation learning (2020, methodology): “We use the standard split of 1,200, 100, and 670 videos for training, validation, and testing (Liu et al., 2019; Venugopalan et al., 2015b; Xu et al., 2015).”
Learning Joint Embedding with Multimodal Cues for Cross-Modal Video-Text Retrieval (2018, methodology): “Recently, a few methods have been proposed for learning visualsemantic embeddings for the video-language matching task [6, 24, 25, 30, 36].”
Dual Encoding for Zero-Example Video Retrieval (2018, methodology): “Different from [24,34], we propose dual multi-level encoding for both videos and queries in advance to common space learning.”
Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddings (2019, background): “Standard examples include building a joint embedding for images and text [11, 36], videos and audio [33] and, more related to our work, for videos and action labels [15], videos and text [8, 14, 40] or some of those combined [25, 24, 22].”
A Survey of Multi-View Representation Learning (2016, unknown): “[191] propose a unified framework that jointly models video and the corresponding text sentence.”
Predicting Visual Features From Text for Image and Video Caption Retrieval (2017, methodology): “The word embedding matrix used in [8], [31], [43], [47] is instantiated by a word2vec model pre-trained on large-scale text corpora.”