A Multi-scale Multiple Instance Video Description Network

By Huijuan Xu, Subhashini Venugopalan, Vasili Ramanishka, Marcus Rohrbach, Kate Saenko. 2015

64 citations recorded by the source corpus.

Generating natural language descriptions for in-the-wild videos is a challenging task. Most state-of-the-art methods for solving this problem borrow existing deep convolutional neural network (CNN) architectures (AlexNet, GoogLeNet) to extract a visual representation of the input video. However, these deep CNN architectures are designed for single-label centered-positioned object classification. While they generate strong semantic features, they have no inherent structure allowing them to detect multiple objects of different sizes and locations in the frame. Our paper tries to solve this problem by integrating the base CNN into several fully convolutional neural networks (FCNs) to form a multi-scale network that handles multiple receptive field sizes in the original image. FCNs, previously applied to image segmentation, can generate class heat-maps efficiently compared to sliding window mechanisms, and can easily handle multiple scales. To further handle the ambiguity over multiple objects and locations, we incorporate the Multiple Instance Learning mechanism (MIL) to consider objects in different positions and at different scales simultaneously. We integrate our multi-scale multi-instance architecture with a sequence-to-sequence recurrent neural network to generate sentence descriptions based on the visual representation. Ours is the first end-to-end trainable architecture that is capable of multi-scale region processing. Evaluation on a Youtube video dataset shows the advantage of our approach compared to the original single-scale whole frame CNN model. Our flexible and efficient architecture can potentially be extended to support other video processing tasks.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks (2015, methodology): “To avoid this issue, more sophisticated ways of encoding video features were proposed in later work, using either a recurrent encoder [11, 46, 53] or an attention model [55].”
Trainable performance upper bounds for image and video captioning (2015, methodology): “…on COCO image captioning has been improved rapidly from 0.17 to 0.31 in BLEU with a series of work by Kiros et al. (2014); Devlin et al. (2015b); Donahue et al. (2015); Vinyals et al. (2014); Xu et al. (2015b); Mao et al. (2015); Karpathy & Fei-Fei (2014); Bengio et al. (2015); Qi Wu et al. (2015).”
EEGtoText: Learning to Write Medical Reports from EEG Recordings (2019, background): “Caption Generation There has been a large body of works that build connection between visual data (e.g., images or videos) and textual data (e.g., summary or caption) in computer vision domain(Vinyals et al., 2015; Xu et al., 2015b; Rennie et al., 2017; Zhou et al., 2017).”
Dense-Captioning Events in Videos (2017, background): “To avoid this issue, others have proposed either a recurrent encoder [6, 37, 42] or an attention mechanism [48].”