Improving LSTM-based Video Description with Linguistic Knowledge Mined from Text

By Subhashini Venugopalan, Lisa Anne Hendricks, Raymond J. Mooney, Kate Saenko. 2016

122 citations recorded by the source corpus.

This paper investigates how linguistic knowledge mined from large text corpora can aid the generation of natural language descriptions of videos. Specifically, we integrate both a neural language model and distributional semantics trained on large text corpora into a recent LSTM-based architecture for video description. We evaluate our approach on a collection of Youtube videos as well as two large movie description datasets showing significant improvements in grammaticality while modestly improving descriptive quality.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Unsupervised Pretraining for Sequence to Sequence Learning (2016, methodology): “To be even more efficient, the softmax of the decoder is initialized with the softmax of the pretrained target side LM.”
Reinforced Video Captioning with Entailment Rewards (2017, methodology): “…sequence-to-sequence models for video captioning, using attention, hierarchical RNNs, 3D-CNN video features, joint embedding spaces, language fusion, etc., but using word-level cross entropy loss training (Venugopalan et al., 2015a; Yao et al., 2015; Pan et al., 2016a,b; Venugopalan et al., 2016).”
Learning to Compose Topic-Aware Mixture of Experts for Zero-Shot Video Captioning (2018, background): “As a result, it has attracted increasing attention in recent years in both NLP (Venugopalan et al. 2016; Wang et al. 2018a) and computer vision communities (Krishna et al. 2017).”
Multimodal Machine Learning: A Survey and Taxonomy (2017, methodology): “, using autoencoder models [12], [83]) or supervised data from a different but related domain [9], [221].”
Augmenting Neural Machine Translation with Knowledge Graphs (2019, methodology): “Differently from Venugopalan et al. (2016), we maximize the vector values of entities found in the bilingual corpora with the values of entities’ labels from the KGE.”
Captioning Images with Diverse Objects (2016, background): “While the final transformation layer (W glove) reverses the mapping [15, 25] of a dense vector back to the full vocabulary with the help of a softmax activation function.”
A review on the long short-term memory model (2020, background): “For example, having a CNN feed a sequence of frames to LSTM layers to generate the word sequence, where (Venugopalan et al. 2016) integrated linguistic knowledge from large text corpora.”
Multi-Task Video Captioning with Video and Entailment Generation (2017, background): “More recently, Venugopalan et al. (2016) explored linguistic improvements to the caption decoder by fusing it with external language models.”
Improving Event Extraction via Multimodal Integration (2017, unknown): “[12, 31, 38] propose approaches of image/video captioning.”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, background): “Further, sequence-to-sequence based approaches are improved with external corpora (Venugopalan et al., 2016) and also attention with various techniques such as soft-attention (Yao et al., 2015), multimodal fusion (Hori et al., 2017), temporal attention (Song et al., 2017), semantic consistency (Gao…”