Learning Language-Visual Embedding for Movie Understanding with Natural-Language

By Atousa Torabi, Niket Tandon, Leonid Sigal. 2016

110 citations recorded by the source corpus.

Learning a joint language-visual embedding has a number of very appealing properties and can result in variety of practical application, including natural language image/video annotation and search. In this work, we study three different joint language-visual neural network model architectures. We evaluate our models on large scale LSMDC16 movie dataset for two tasks: 1) Standard Ranking for video annotation and retrieval 2) Our proposed movie multiple-choice test. This test facilitate automatic evaluation of visual-language models for natural language video annotation based on human activities. In addition to original Audio Description (AD) captions, provided as part of LSMDC16, we collected and will make available a) manually generated re-phrasings of those captions obtained using Amazon MTurk b) automatically generated human activity elements in "Predicate + Object" (PO) phrases based on "Knowlywood", an activity knowledge mining model. Our best model archives Recall@10 of 19.2% on annotation and 18.9% on video retrieval tasks for subset of 1000 samples. For multiple-choice test, our best model achieve accuracy 58.11% over whole LSMDC16 public test-set.

Subfields: Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, methodology): “Our off-the-shelf HowTo100M model outperforms [22, 24, 53, 64, 65] that are directly trained on MSR-VTT.”
All in One: Exploring Unified Video-Language Pre-Training (2022, methodology): “Datasets: In this task, we adopt MSRVTT multiple-choice test set [40] and LSMDC multiple-choice test set [33].”
VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling (2021, methodology): “Ended TGIF-Frame [6] 1,540 MSRVTT-QA [7] 1,500 MSVD-QA [88] 1,000 LSMDC-FiB [81] 908”
MERLOT: Multimodal Neural Script Knowledge Models (2021, methodology): “The tasks we report on are: TVQA [53], TVQA(+) [54], VLEP [55], MSRVTT-QA [105], MSRVTTMultichoice [109], LSMDC-Multichoice, LSMDC fill-in-the-blank QA [97, 81], ActivityNetQA [110, 37], TGIFQA [44], and DramaQA [18]5.”
ActBERT: Learning Global-Local Video-Text Representations (2020, neutral): “For multi-choice VideoQA, we ﬁne-tune the pre-trained ActBERT on the Method Accuracy LSTM-fusion [49] 38.3 C+LSTM+SA-FC7 [39] 60.2 VSE-LSTM [14] 67.3 SNUVL [50] 65.4 EITanque [11] 65.5 CT-SAN [51] 66.4 MLB [13] 76.1 JSFusion [49] 83.4 ActBERT 85.7 Table 5: Video question answering…”
A Joint Sequence Fusion Model for Video Question Answering and Retrieval (2018, methodology): “[1] extend the LSMDC movie description dataset to the VQA domain, introducing several new tasks such as multiplechoice [12] and fill-in-the-blank [34].”
Movie Description (2016, background): “Both datasets are similar in that they represent web-videos, while our proposed datasets focus on movies.”
Localizing Moments in Video with Natural Language (2017, methodology): “[48] for video description and [46] for whole video retrieval).”
Learning a Text-Video Embedding from Incomplete and Heterogeneous Data (2018, methodology): “Finally, we also signiﬁcantly outperform the C+LSTM+SA+FC7 [56] baseline that augments the MPII movie dataset with COCO image caption data.”
Learning Joint Embedding with Multimodal Cues for Cross-Modal Video-Text Retrieval (2018, methodology): “Recently, a few methods have been proposed for learning visualsemantic embeddings for the video-language matching task [6, 24, 25, 30, 36].”
VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding (2021, neutral): “Accuracy ↑ S UPERVISED LSTM-fusion (Yu et al., 2018) 38.3 C+LSTM+SA-FC7 (Torabi et al., 2016) 60.2 SNUVL (Yu et al., 2016) 65.4 EITanque (Kaufman et al., 2017) 65.5 CT-SAN (Yu et al., 2017) 66.4 VSE-LSTM (Kiros et al., 2014) 67.3 MLB (Kim et al., 2016) 76.1 JSFusion(Yu et al., 2018) 83.4…”
CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval (2021, methodology): “The pixel-level approach trains the model with raw video as input directly (Torabi et al., 2016; Kiros et al., 2014; Yu et al., 2016a; Kaufman et al., 2017; Yu et al., 2017, 2018).”
Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddings (2019, methodology): “With the advent of neural networks, works shifted to use RNNs, Gated Recurrent Units (GRU) or Long Short-Term Memory (LSTM) units to extract textual features [8] or to use these models within the embedding network [15, 18, 24, 25, 34] for both modalities.”
Measuring Social Biases in Grounded Vision and Language Embeddings (2020, methodology): “While we discuss joint vision and language embeddings, the methods introduced here apply to any grounded embeddings, such as joint audio and lan-Number Number of statistically significant tests out of 7 total race bias tests guage embeddings (Kiela and Clark, 2015; Torabi et al., 2016).”
TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignment (2021, background): “Some works have relied on attention mechanisms to extract key information from videos [44, 54], while others preserve visual information by composing pairwise joint representation using 3D tensors [52] or use multi-level video encoders to separately encode the spatial and temporal cues [11].”
Deep Multimodal Feature Encoding for Video Ordering (2020, methodology): “However, as the LSMDC dataset already comes with pre-segmented clips, we redefine the problem as grouping clips into unique scenes.”
Multimodal Abstractive Summarization for How2 Videos (2019, background): “With the abundance of videos online, there has been an increase in demand for efﬁcient ways to search and retrieve relevant videos (Song et al., 2011; Wang et al., 2012; Otani et al., 2016; Torabi et al., 2016).”