Deep Multimodal Feature Encoding for Video Ordering

By Vivek Sharma, Makarand Tapaswi, Rainer Stiefelhagen. 2020

11 citations recorded by the source corpus.

True understanding of videos comes from a joint analysis of all its modalities: the video frames, the audio track, and any accompanying text such as closed captions. We present a way to learn a compact multimodal feature representation that encodes all these modalities. Our model parameters are learned through a proxy task of inferring the temporal ordering of a set of unordered videos in a timeline. To this end, we create a new multimodal dataset for temporal ordering that consists of approximately 30K scenes (2-6 clips per scene) based on the "Large Scale Movie Description Challenge". We analyze and evaluate the individual and joint modalities on three challenging tasks: (i) inferring the temporal ordering of a set of videos; and (ii) action recognition. We demonstrate empirically that multimodal representations are indeed complementary, and can play a key role in improving the performance of many applications.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

SplitNN-driven Vertical Partitioning (2020, methodology): “One can readily employ other encoding methods like Compact Bilinear Pooling (Diba et al. , 2017; Gao et al. , 2016), Temporal Compact Bilinear Pooling (Sharma et al. , 2020), NetVLAD (Arandjelovi ´ c et al. , 2016) instead of the pooling mechanisms for a more robust representation learning.”
Look Before you Speak: Visually Contextualized Utterances (2020, background): “A large number of multimodal architectures focus on late fusion of modalities, with popular choices being summation, concatenation and canonical correlation analysis [9, 74, 78].”