Slow and Steady Feature Analysis: Higher Order Temporal Coherence in Video

By Dinesh Jayaraman, Kristen Grauman. 2015

153 citations recorded by the source corpus.

How can unlabeled video augment visual learning? Existing methods perform "slow" feature analysis, encouraging the representations of temporally close frames to exhibit only small differences. While this standard approach captures the fact that high-level visual signals change slowly over time, it fails to capture how the visual content changes. We propose to generalize slow feature analysis to "steady" feature analysis. The key idea is to impose a prior that higher order derivatives in the learned feature space must be small. To this end, we train a convolutional neural network with a regularizer on tuples of sequential frames from unlabeled video. It encourages feature changes over time to be smooth, i.e., similar to the most recent changes. Using five diverse datasets, including unlabeled YouTube and KITTI videos, we demonstrate our method's impact on object, scene, and action recognition tasks. We further show that our features learned from unlabeled video can even surpass a standard heavily supervised pretraining approach.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Self-Supervised Learning of Video-Induced Visual Invariances (2019, background): “Slow and steady feature analysis [30] attempts to learn representations that exhibit higher order temporal coherence.”
Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning (2016, background): “We show through numerous experiments that this combination leads to a competitive semi-supervised learning method.”
Self-labelling via simultaneous clustering and representation learning (2019, background): “…audio-video alignment (Arandjelovi´c & Zisserman, 2017), ego-motion estimation (Jayaraman & Grauman, 2015), slow feature analysis with higher order temporal coherence (Jayaraman & Grauman, 2016), transformation between frames (Agrawal et al., 2015) and patch tracking in videos (Wang & Gupta, 2015).”
Unsupervised Representation Learning by Sorting Sequences (2017, background): “Examples include enforcing the temporal smoothness of representation throughout a video [26, 15], applying tracking to capture appearance variation of moving objects [42], learning transformation in ego-motion videos [14, 2], verifying the order of input sequence [25, 9], and the transformation betw”
Video Representation Learning by Dense Predictive Coding (2019, methodology): “[11, 13, 43] use the temporal coherence as a proxy loss.”
Cross Pixel Optical Flow Similarity for Self-Supervised Learning (2018, neutral): “n recognition. Temporal context is a powerful signal. [8,9,10] learn to predict the correct ordering of frames. [11] exploit both temporal and spatial co-occurrence statistics to learn visual groups. [12] extend slow feature analysis using higher order temporal coherence. [13] track patches in a vid”
Object-Centric Representation Learning from Unlabeled Videos (2016, methodology): “Most prior work in this space produces a holistic image embedding, and attempts to learn feature representations for video frames as a whole [6,20,9,8,10].”
Unsupervised Learning using Sequential Verification for Action Recognition (2016, background): “Moving beyond just temporal smoothness, [53] enforces additional ‘steadiness’ constraints on the features so that the change of features across frames is meaningful.”
Self-Supervised Learning via Conditional Motion Propagation (2019, background): “[12] train a CNN with a regularizer that feature changes over time should be smooth.”
ShapeCodes: Self-supervised Feature Learning by Lifting Views to Viewgrids (2017, background): “Our work relates to a growing body of work on self-supervised representation learning [7, 8, 9, 10, 11, 12, 13, 14, 15, 16], where “pretext” tasks requiring no manual labels are posed to the feature learner to instill visual concepts useful for recognition.”
Structured Images for RGB-D Action Recognition (2017, background): “For example, one can conventionally consider a video as a sequence of still images [40] with some form of temporal smoothness [27, 15] and feed them into a ConvNet, or extend ConvNet to a third, temporal dimension [16, 30] by replacing 2D filters with 3D ones, or regard the video as the output of a ”
Deep Feature Flow for Video Recognition (2016, methodology): “The similarity is even stronger in the deep feature maps, which encode high level semantic concepts [45, 21].”
Ego-Topo: Environment Affordances From Egocentric Video (2020, methodology): “Our localization network draws inspiration from the retrieval network employed in [59] to build maps for embodied agent navigation, and more generally prior work leveraging temporal coherence to self-supervise image similarity [24, 49, 32].”
Teaching Compositionality to CNNs (2017, background): “Examples include learning representations from video sequences [2, 10, 17], encouraging the utilization of depth information [14], and using physical interaction with the eninput images”
Deep Multimodal Feature Encoding for Video Ordering (2020, methodology): “It is also worth noting that temporal ordering has been used for other applications such as predicting what happens next in egocentric videos [70], using the steadiness of visual change in videos to penalize higher order differences [21], or to recognize complex, long-term activities [26].”
Self-Supervised Learning by Cross-Modal Audio-Video Clustering (2019, background): “Other pretext tasks for video modeling include temporal frame ordering [9, 34, 38, 72], establishing region or object correspondences across frames [23, 24, 69, 70], predicting ﬂow [31] or colors [66], as well as tracking [71].”
Shuffle and Learn: Unsupervised Learning Using Temporal Order Verification (2016, background): “Moving beyond just temporal smoothness, [37] enforces additional ‘steadiness’ constraints on the features so that the change of features across frames is meaningful.”