Learning and Using the Arrow of Time

By Donglai Wei, Joseph J. Lim, Andrew Zisserman, William T. Freeman. 2018

363 citations recorded by the source corpus.

We seek to understand the arrow of time in videos - what makes videos look like they are playing forwards or backwards? Can we visualize the cues? Can the arrow of time be a supervisory signal useful for activity analysis? To this end, we build three large-scale video datasets and apply a learning-based approach to these tasks. To learn the arrow of time efficiently and reliably, we design a ConvNet suitable for extended temporal footprints and for class activation visualization, and study the effect of artificial cues, such as cinematographic conventions, on learning. Our trained model achieves state-of-the-art performance on large-scale real-world video datasets. Through cluster analysis and localization of important regions for the prediction, we examine learned visual cues that are consistent among many samples and show when and where they occur. Lastly, we use the trained ConvNet for two applications: self-supervision for action recognition, and video forensics - determining whether Hollywood film clips have been deliberately reversed in time, often used as special effects.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Self-Supervised Learning of Video-Induced Visual Invariances (2019, background): “As a result, and with the recent emergence of research video data sets [1, 59], videos have been investigated in the context of self-supervision (for example, [40, 64, 63, 28, 65, 73, 41, 51, 42, 3, 2]).”
Learning Video Representations from Textual Web Supervision (2020, methodology): “For example, prior approaches have successfully lever-aged supervision from clip and frame ordering [47, 16, 38, 66, 68, 33], geometry [17, 29], motion [53, 37], colorization [62], cycle consistency [14, 64], video prediction [44, 42, 61, 60, 63], tempo [69], and in-video consistency [22, 54].”
Self-Supervised Learning of Pretext-Invariant Representations (2019, background): “For video data, these pretext tasks learn representations by ordering video frames [1, 18, 34, 41, 51, 79, 83], tracking [62, 77], or using cross-modal signals like audio [2, 3, 19, 36, 60, 61].”
A critical analysis of self-supervision, or what we can learn from a single image (2019, background): “Typical pretext tasks are based on temporalcontext (Misra et al., 2016; Wei et al., 2018; Lee et al., 2017; Sermanet et al., 2018), spatio-temporal”
MERLOT: Multimodal Neural Script Knowledge Models (2021, background): “In addition to extrapolation, past work has studied deshufﬂing objectives in videos [82, 115], though this has mostly been limited to the visual modality.”
Self-labelling via simultaneous clustering and representation learning (2019, background): “Typical pretext tasks are based on temporal-context (Misra et al., 2016; Wei et al., 2018; Lee et al., 2017; Sermanet et al., 2018), spatio-temporal cues (Isola et al.”
Transformers in Vision: A Survey (2021, methodology): “The context-based pretext methods solve problems such as a jigsaw puzzle on image patches [5, 101, 150], masked object classiication [182], predict geometric transformation such as rotation [67, 94], or verify temporal sequence of video frames [114, 144, 218].”
Audio-Visual Scene Analysis with Self-Supervised Multisensory Features (2018, background): “predicting the temporal ordering in a video [17,18,19].”
Video Representation Learning by Dense Predictive Coding (2019, background): “However, ConvNets are notorious for learning shortcuts for tackling tasks [4, 28, 45].”
Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction. (2018, methodology): “In this section, we compare our 3DRotNet with other self-supervised methods on action recognition task including the 2DCNN-based [13, 29, 30, 47, 49] and the 3DCNN-based methods [43, 22].”
Cross Pixel Optical Flow Similarity for Self-Supervised Learning (2018, neutral): “[8,9,10] learn to predict the correct ordering of frames.”
Noise Estimation Using Density Estimation for Self-Supervised Multimodal Learning (2020, background): “…self-supervised machine learning (Zhang, Isola, and Efros 2016; Noroozi and Favaro 2016; Pathak et al. 2016; Misra, Zitnick, and Hebert 2016; Wei et al. 2018; Vondrick, Pirsiavash, and Torralba 2016; Srivastava, Mansimov, and Salakhudinov 2015) presents new ways to better utilize the…”
Video Understanding as Machine Translation (2020, methodology): “Such approaches typically define pretext tasks that transform the video in a certain way, and train the model to predict that transformation [33, 57, 4].”
Hard Negative Mixing for Contrastive Learning (2020, background): “…the sequential nature of the temporal dimension, e.g . future frame prediction and reconstruction [60], shufﬂing and then predicting or verifying the order of frames or clips [47, 38, 79], predicting the ”arrow of time” [74], pace [72] or predicting the “odd” element [21] from a set of clips.”
MAST: A Memory-Augmented Self-Supervised Tracker (2020, background): “[14, 43, 66] exploit spatio-temporal ordering for learning video representations.”
Self-supervised Spatiotemporal Representation Learning by Exploiting Video Continuity (2021, background): “2020), arrow of time (Wei et al. 2018), motion and appearance statistics (Wang et al.”
Self-Supervised Video Representation Learning with Space-Time Cubic Puzzles (2018, methodology): “In particular, we compare with (Mobahi, Collobert, and Weston 2009; Wang and Gupta 2015; Misra, Zitnick, and Hebert 2016; Lee et al. 2017; Gan et al. 2018; Wei et al. 2018) using the RGB video data.”
Multi-Modal Domain Adaptation for Fine-Grained Action Recognition (2020, background): “Self-supervision methods learn representations from the temporal [13, 59] and multi-modal structure of video [1, 25], leveraging pretraining on a large corpus of unlabelled videos.”
Oops! Predicting Unintentional Action in Video (2019, methodology): “Various types of self-supervised signals have been used to learn strong visual representations, such as spatial arrangement [44], contextual information [14, 68], color [37, 64], the arrow of time [70, 75, 23, 38, 41, 17], future prediction [40, 63, 76, 45], consistency inmotion [3, 26], viewsynthes”