Video Captioning with Transferred Semantic Attributes

By Yingwei Pan, Ting Yao, Houqiang Li, Tao Mei. Computer Vision and Pattern Recognition, 2016

340 citations recorded by the source corpus.

Automatically generating natural language descriptions of videos plays a fundamental challenge for computer vision community. Most recent progress in this problem has been achieved through employing 2-D and/or 3-D Convolutional Neural Networks (CNNs) to encode video content and Recurrent Neural Networks (RNNs) to decode a sentence. In this paper, we present Long Short-Term Memory with Transferred Semantic Attributes (LSTM-TSA)—a novel deep architecture that incorporates the transferred semantic attributes learnt from images and videos into the CNN plus RNN framework, by training them in an end-to-end manner. The design of LSTM-TSA is highly inspired by the facts that 1) semantic attributes play a significant contribution to captioning, and 2) images and videos carry complementary semantics and thus can reinforce each other for captioning. To boost video captioning, we propose a novel transfer unit to model the mutually correlated attributes learnt from images and videos. Extensive experiments are conducted on three public datasets, i.e., MSVD, M-VAD and MPII-MD. Our proposed LSTM-TSA achieves to-date the best published performance in sentence generation on MSVD: 52.8% and 74.0% in terms of BLEU@4 and CIDEr-D. Superior results are also reported on M-VAD and MPII-MD when compared to state-of-the-art methods.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/0d3b5ffff118326fea73341a86a7c29423eb95f0

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Multi-level Attention Networks for Visual Question Answering (2017, background): “Previous works on image/video captioning [6, 22, 23, 35, 36] and visual question answering [30, 31] have shown that extracting explicit high-level”
Video Captioning With Attention-Based LSTM and Semantic Consistency (2017, background): “Due to the development of Recurrent Neural Network (RNN) [29] and LSTM, researchers have striven to automatically describe an image/video with a correct and novel natural language sentence [30], [31], [20], [32], [33], [34], [35], [36], [37], [38].”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, methodology): “from attention based methods, novel architectures are also explored such as incorporation of semantic attributes learned from videos (Pan et al., 2017), ensemble based description generator networks (Shetty et al.”
BBS-Net: RGB-D Salient Object Detection with a Bifurcated Backbone Strategy Network (2020, background): “better understand the contextual and semantic information to generate high-quality saliency maps. Besides, since imagebased SOD may be limited in some real-world applications such as video captioning [55], autonomous driving [56] and robotic interaction [57], SOD algorithms [8], [9] have also been e”
Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning (2023, background): “Dense video captioning lies at the intersection of event localization [25, 29, 33, 62, 65, 66, 84, 127] and event captioning [30,63,75,97,104].”
End-to-End Dense Video Captioning with Masked Transformer (2018, neutral): “, via mean-pooling [31, 10], recurrent nets [7, 30], and attention mechanisms [35, 22, 10].”
Pointing Novel Objects in Image Captioning (2019, positive): “Specifically, inspired from the sequence learning models in image/video captioning [6, 11, 18, 19, 28, 31, 32] and copying mechanism [33], we equip the regular CNN plus RNN language model with the copying layer, which predicts each target word through not only the word-byword generation by LSTM-base”
Boosting Image Captioning with Attributes (2016, positive): “cesses of probabilistic sequence models leveraged in statistical machine translation [27] and image/video captioning [20, 21, 31], we aim to formulate our image captioning models in an end-to-end fashion based on RNNs which encode the given image and/or its detected attributes into a fixed dimension”
Action Modifiers: Learning From Adverbs in Instructional Videos (2019, background): “Several problems are at the intersection between language and video: captioning [24, 38, 55, 59], retrieval [9, 16, 21, 31, 33, 52, 54] and visual question answering [15, 56, 57, 61].”
Image Captioning and Visual Question Answering Based on Attributes and External Knowledge (2016, methodology): “Following our proposed attributes-based framework [17], several recent works [25], [26], [27] have shown that ‘attributes’ or ‘high-level concepts’”
Grounded Video Description (2018, background): “Visual attention usually comes in the form of temporal attention [34] (or spatial-attention [32] in the image domain), semantic attention [13, 35, 36, 41] or both [19].”
Exploring Visual Relationship for Image Captioning (2018, methodology): “Inspired by the recent successes of sequence models leveraged in image/video captioning [26,27,34] and region-level attention mechanism [2,8], we aim to formulate our image captioning model in a R-CNN plus RNN scheme.”
Joint Syntax Representation Learning and Visual Cue Translation for Video Captioning (2019, background): “Most existing methods of video captioning [48, 41, 13, 30, 4, 12, 8, 1] mainly focus on investigating various visual perception models by exploiting informative semantics without considering learning of syntax representation for generating sentences.”
Controllable Video Captioning With POS Sequence Guidance Based on Gated Fusion Network (2019, methodology): “In this subsection, we compare our method with the state-of-the-art methods with multiple features on benchmark datasets, including SA [53], M3 [47], v2t navigator [18], Aalto [36], VideoLab [31], MALSTM [51], M&M-TGM [4], PickNet [8], LSTMTSAIV [28], SibNet [23], MGSA [5], and SCNLSTM [14], most of”
Visual to Text: Survey of Image and Video Captioning (2019, methodology): “S2VT [102], HRNE [107] and LSTM-BA [108]; (2) the spatialtemporal CNN based encoding methods including SA [103], h-RNN [106], LSTM-E [107] and LSTM-TSA [109].”
Reconstruction Network for Video Captioning (2018, methodology): “However, as stated in [25], the aforementioned approaches highly depend on the prede-ﬁned template and are thus limited by the ﬁxed syntactical structure, which is inﬂexible for sentence generation.”
Syntax-Aware Action Targeting for Video Captioning (2020, methodology): “On top of that, methods such as conditional random ﬁelds (CRF) [12] and recurrent neural networks (RNNs) [12, 52] are proposed to replace the original mean-pooling in the encoder, and attention mechanisms [15, 35, 62, 28] were incorporated to extract salient frames or regions in decoding phase.”
SibNet: Sibling Convolutional Encoder for Video Captioning (2018, methodology): “Besides, both LSTM-TSA [27] and SCN [9] rely on hundreds of datasetspecific “semantic attributes”, which are manually selected from thousands of candidates.”
Learning Multimodal Attention LSTM Networks for Video Captioning (2017, methodology): “Research in this direction mainly includes two different directions: template-based language methods [3, 7, 10, 17, 28] and sequence learning approaches [6, 13–15, 24, 30, 33].”