Jointly Modeling Embedding and Translation to Bridge Video and Language

By Yingwei Pan, Tao Mei, Ting Yao, Houqiang Li, Yong Rui. 2015

546 citations recorded by the source corpus.

Automatically describing video content with natural language is a fundamental challenge of computer vision. Re-current Neural Networks (RNNs), which models sequence dynamics, has attracted increasing attention on visual interpretation. However, most existing approaches generate a word locally with the given previous words and the visual content, while the relationship between sentence semantics and visual content is not holistically exploited. As a result, the generated sentences may be contextually correct but the semantics (e.g., subjects, verbs or objects) are not true. This paper presents a novel unified framework, named Long Short-Term Memory with visual-semantic Embedding (LSTM-E), which can simultaneously explore the learning of LSTM and visual-semantic embedding. The former aims to locally maximize the probability of generating the next word given previous words and visual content, while the latter is to create a visual-semantic embedding space for enforcing the relationship between the semantics of the entire sentence and visual content. The experiments on YouTube2Text dataset show that our proposed LSTM-E achieves to-date the best published performance in generating natural sentences: 45.3% and 31.0% in terms of BLEU@4 and METEOR, respectively. Superior performances are also reported on two movie description datasets (M-VAD and MPII-MD). In addition, we demonstrate that LSTM-E outperforms several state-of-the-art techniques in predicting Subject-Verb-Object (SVO) triplets.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Self-Supervised MultiModal Versatile Networks (2020, background): “With larger datasets [48, 69, 71, 92, 96], many works have profited from learning such a joint visual-textual space [17, 19, 29, 30, 41, 53, 57, 63, 68, 84, 85, 87, 88, 93].”
End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, background): “Numerous prior works [19, 21, 27, 28, 42, 51, 55, 60, 62, 88, 79, 80, 83, 84] have used image / video description datasets [48, 63, 65, 87, 90] to learn an embed-”
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, methodology): “A common approach to model vision and language is learning a joint embedding space where visual and textual cues are adjacent if and only if they are semantically similar [6, 8, 10, 11, 25, 32, 35, 37, 38, 59, 54, 55, 57].”
ActBERT: Learning Global-Local Video-Text Representations (2020, neutral): “The dominant approach for multi-modal modeling is to leverage Recurrent Neural Networks (RNNs) and their variants, e.g ., Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU), to model sequence relations, e.g ., [29, 55].”
Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training (2020, background): “Another popular task is video captioning (Venugopalan et al., 2015; Pan et al., 2016; Gan et al., 2017; Zhou et al., 2018b, 2019), mostly benchmarking on Youtube2Text (Guadarrama et al., 2013), MSR-VTT (Xu et al., 2016), YouCook2 (Zhou et al., 2018a), ActivityNet Captions (Krishna et al., 2017) and…”
Dense-Captioning Events in Videos (2017, positive): “To elevate the lack of detail from existing action detection models, subsequent work has explored explaining video semantics using sentence descriptions [29, 32, 28, 38, 37].”
Localizing Moments in Video with Natural Language (2017, background): “Contemporary models integrate deep video representations with recurrent language models [25, 31, 47, 48, 56].”
TGIF: A New Dataset and Benchmark on Animated GIF Description (2016, background): “This process is error-prone and the errors are particularly harmful to image sequence modeling because a few irrelevant frames either at the beginning or the end of a sequence can significantly alter the sequence representation.”
Improving Interpretability of Deep Neural Networks with Semantic Information (2017, methodology): “A key difference from previous works [32, 34, 26] which use CNN features as video representations is that we stack a bi-directional LSTM model [29] on top of a CNN model to characterize the video temporal variation in both input directions.”
Hierarchical Recurrent Neural Encoder for Video Representation with Application to Captioning (2015, methodology): “The same as [9], other works such as [38, 42, 21] utilize the LSTM essentially as a recurrent neural network language model to generate video descriptions, which conditions on either the average pooled frame-level features or the context vector linearly blended by the attention mechanism [1].”
Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks (2015, methodology): “We compare our approach (h-RNN) on YouTubeClips with six state-of-the-art methods: LSTM-YT [48], S2VT [47], MM-VDN [54], TA [56], and LSTM-E [32].”
A Closer Look at Spatiotemporal Convolutions for Action Recognition (2017, background): “3D CNNs features were also demonstrated to generalize well to other tasks, including action detection [28], video captioning [24], and hand gesture detection [23].”
Learning a Text-Video Embedding from Incomplete and Heterogeneous Data (2018, background): “Towards this goal recent works study representations of vision and language addressing tasks such as visual question answering [1], [2], action learning and discovery [3], [4], [5], text-based event localization [6] as well as video captioning, retrieval and summarization [7], [8], [9], [10].”
Exploring Visual Relationship for Image Captioning (2018, methodology): “Inspired by the recent successes of sequence models leveraged in image/video captioning [26,27,34] and region-level attention mechanism [2,8], we aim to formulate our image captioning model in a R-CNN plus RNN scheme.”
Semantic Compositional Networks for Visual Captioning (2016, background): “Most of these approaches learn a probabilistic model of the caption, conditioned on an image or a video [29, 47, 13, 20, 48, 52, 10, 46, 32, 56].”
Learning Joint Embedding with Multimodal Cues for Cross-Modal Video-Text Retrieval (2018, methodology): “We report score for JMET from [6].”
Boosting Image Captioning with Attributes (2016, positive): “cesses of probabilistic sequence models leveraged in statistical machine translation [27] and image/video captioning [20, 21, 31], we aim to formulate our image captioning models in an end-to-end fashion based on RNNs which encode the given image and/or its detected attributes into a fixed dimension”