Self-supervised Video Representation Learning for Correspondence Flow

By Zihang Lai, Weidi Xie. 2019

48 citations recorded by the source corpus.

The objective of this paper is self-supervised learning of feature embeddings that are suitable for matching correspondences along the videos, which we term correspondence flow. By leveraging the natural spatial-temporal coherence in videos, we propose to train a ``pointer'' that reconstructs a target frame by copying pixels from a reference frame. We make the following contributions: First, we introduce a simple information bottleneck that forces the model to learn robust features for correspondence matching, and prevent it from learning trivial solutions, \eg matching based on low-level colour information. Second, to tackle the challenges from tracker drifting, due to complex object deformations, illumination changes and occlusions, we propose to train a recursive model over long temporal windows with scheduled sampling and cycle consistency. Third, we achieve state-of-the-art performance on DAVIS 2017 video segmentation and JHMDB keypoint tracking tasks, outperforming all previous self-supervised learning approaches by a significant margin. Fourth, in order to shed light on the potential of self-supervised learning on the task of video correspondence flow, we probe the upper bound by training on additional data, \ie more diverse videos, further demonstrating significant improvements on video segmentation.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Video Representations from Textual Web Supervision (2020, background): “For example, prior approaches have successfully leveraged supervision from clip and frame ordering [47, 16, 38, 66, 68, 33], geometry [17, 29], motion [53, 37], colorization [62], cycle consistency [14, 64], video prediction [44, 42, 61, 60, 63], tempo [69], and in-video consistency [22, 54].”
Space-Time Correspondence as a Contrastive Random Walk (2020, unknown): “Following common practice [88, 43, 44], we use a queue of context frames and restrict the set of source nodes considered to be within a spatial neighborhood of the query node for efficiency.”
Self-Supervised Learning by Cross-Modal Audio-Video Clustering (2019, background): “Other pretext tasks for video modeling include temporal frame ordering [9, 34, 38, 72], establishing region or object correspondences across frames [23, 24, 69, 70], predicting flow [31] or colors [66], as well as tracking [71].”
Oops! Predicting Unintentional Action in Video (2019, methodology): “Various types of self-supervised signals have been used to learn strong visual representations, such as spatial arrangement [42], contextual information [12, 66], color [35, 62], the arrow of time [68, 73, 21, 36, 39, 15], future prediction [38, 61, 74, 43], consistency in motion [3, 24], view synth”
Learning Video Object Segmentation From Unlabeled Videos (2020, methodology): “We train the whole network from scratch on the OxUvA[51] tracking dataset, as in[22].”