Learning Temporal Embeddings for Complex Video Analysis

By Vignesh Ramanathan, Kevin D. Tang, Greg Mori, Li Fei-Fei. 2015

72 citations recorded by the source corpus.

In this paper, we propose to learn temporal embeddings of video frames for complex video analysis. Large quantities of unlabeled video data can be easily obtained from the Internet. These videos possess the implicit weak label that they are sequences of temporally and semantically coherent images. We leverage this information to learn temporal embeddings for video frames by associating frames with the temporal context that they appear in. To do this, we propose a scheme for incorporating temporal context based on past and future frames in videos, and compare this to other contextual representations. In addition, we show how data augmentation using multi-resolution samples and hard negatives helps to significantly improve the quality of the learned embeddings. We evaluate various design decisions for learning temporal embeddings, and show that our embeddings can improve performance for multiple video tasks such as retrieval, classification, and temporal order recovery in unconstrained Internet video.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Context Encoders: Feature Learning by Inpainting (2016, methodology): “Consistency across temporal frames has been used as supervision to learn embeddings which perform well on a number of tasks [17, 34].”
Generating Videos with Scene Dynamics (2016, background): “Finally, this paper is related to a growing body of work that capitalizes on large amounts of unlabeled video for visual recognition tasks [18, 47, 37, 13, 24, 25, 3, 32, 26, 27, 19, 41, 42, 1].”
Learning and Using the Arrow of Time (2018, background): “Others have used the temporal ordering of frames as a supervisory signal for learning an embedding [15], for self-supervision training of a ConvNet [6, 13], and for construction of a representation for action recognition [7].”
Connectionist Temporal Modeling for Weakly Supervised Action Labeling (2016, background): “Our goal of understanding the temporal structure of video is related to [10,31,34,37,39,40].”
Object-Centric Representation Learning from Unlabeled Videos (2016, background): “For example, the reconstruction and prediction of a sequence of video frames can serve as the proxy task [26,8].”
Leveraging Weak Semantic Relevance for Complex Video Event Classification (2017, methodology): “To alleviate this issue, methods utilizing semantic information have been proposed [1, 30, 10, 25, 12, 13, 32].”
Learning Temporal Regularity in Video Sequences (2016, background): “[18] trained a network to learn the motion signature of the same temporal coherency prior and used it for event retrieval.”
Unsupervised Learning of Visual Representations using Videos (2015, background): “In another recent preprint, Ramanathan et al. (2015) learn temporal embeddings by extending the Skip-gram model that was proposed for learning word embeddings (Mikolov et al., 2013).”
Form2Fit: Learning Shape Priors for Generalizable Assembly from Disassembly (2019, background): “, θ = 360 20 × argmini j ‖μ i j kit−μ i obj‖(2)2 where j ∈ Z : j ∈ [1,20] and i ∈ Z : i ∈ [1,H×W ].”
3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions (2016, background): “For example, recent works show that the temporal information from videos can be used as a plentiful source of supervision to learn embeddings that are useful for various tasks [14, 27].”
Leveraging Structural Context Models and Ranking Score Fusion for Human Interaction Prediction (2016, background): “The temporal information of a video sequence is also very important for video analysis [8], [9].”
Meta-Amortized Variational Inference and Learning (2019, methodology): “Future work could consider applications of metainference in video prediction (Ramanathan et al. 2015).”
STAT: Spatial-Temporal Attention Mechanism for Video Captioning (2020, methodology): “To begin with, we describe our overall framework, based on the popular ConvNet + LSTM architecture [31]–[37].”