Learning Correspondence From the Cycle-Consistency of Time

By X. Wang, Allan Jabri, Alexei A. Efros. 2019

542 citations recorded by the source corpus.

We introduce a self-supervised method for learning visual correspondence from unlabeled video. The main idea is to use cycle-consistency in time as free supervisory signal for learning visual representations from scratch. At training time, our model learns a feature map representation to be useful for performing cycle-consistent tracking. At test time, we use the acquired representation to find nearest neighbors across space and time. We demonstrate the generalizability of the representation -- without finetuning -- across a range of visual correspondence tasks, including video object segmentation, keypoint tracking, and optical flow. Our approach outperforms previous self-supervised methods and performs competitively with strongly supervised methods.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Video Representations from Textual Web Supervision (2020, methodology): “For example, prior approaches have successfully lever-aged supervision from clip and frame ordering [47, 16, 38, 66, 68, 33], geometry [17, 29], motion [53, 37], colorization [62], cycle consistency [14, 64], video prediction [44, 42, 61, 60, 63], tempo [69], and in-video consistency [22, 54].”
Spatiotemporal Contrastive Video Representation Learning (2020, background): “Video can help supervise the learning of image representations [42, 31, 40, 15, 33], correspondences [43, 8], and robotic behaviors [35] thanks to its rich content about different views of objects and its motion and tracking cues.”
Self-Supervised Visual Representation Learning from Hierarchical Grouping (2020, unknown): “We adopt an approach similar to [45, 19, 44] and randomly sample negative features from the dataset (but operationalized on a per-pixel level).”
Few-Shot Segmentation via Cycle-Consistent Transformer (2021, background): “It is also effective in learning 3D correspondence [49], consistency between video frames [37] and association between different domains [16].”
Kubric: A scalable dataset generator (2022, methodology): “VFS has two key components: first, a self-supervised training phase where the aim is to learn a good similarity metric between points across images, and second, a test-time tracking algorithm based on earlier work [101] which can associate points in unlabeled frames with the points in labeled frames”
PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Tracking (2023, background): “There is also a long line of work which trains directly on unlabelled data, using a variety of auxiliary objectives to encourage tracking to emerge [65, 60, 62, 29, 7].”
CrDoCo: Pixel-Level Domain Transfer With Cross-Domain Consistency (2019, methodology): “In motion analysis, forward-backward consistency check can be used for detecting occlusion [28, 20, 52] or learning visual correspondence [45].”
Self-Supervised Learning by Cross-Modal Audio-Video Clustering (2019, methodology): “ks for video modeling include temporal frame ordering [9,34,38, 72], establishing region or object correspondences across frames [23,24,69,70], predicting ﬂow [31] or colors [66], as well as tracking [71]. Moreover, a large set of selfsupervised approaches on video are built upon the pretext task of”
Self-supervised Co-training for Video Representation Learning (2020, unknown): “motion [1, 14, 31], co-occurrence [30], jigsaw [37], rotation [33], speed prediction [6, 17, 62], future prediction [25, 26, 60], or by temporal coherence [40, 41, 61, 63].”
CoTracker3: Simpler and Better Point Tracking by Pseudo-Labeling Real Videos (2025, unknown): “Accounting for cycle consistency [13, 30, 41] or temporal continuity [9, 43] in videos is another way to obtain a reliable proxy signal to learn correspondences without full supervision, or even to learn generic visual features [10, 40].”
R3M: A Universal Visual Representation for Robot Manipulation (2022, background): “Finally, there is a rich literature of works that study learning image representations from videos [58, 59, 19, 60, 61] outside of the context of robotics.”
Oops! Predicting Unintentional Action in Video (2019, methodology): “Various types of self-supervised signals have been used to learn strong visual representations, such as spatial arrangement [44], contextual information [14, 68], color [37, 64], the arrow of time [70, 75, 23, 38, 41, 17], future prediction [40, 63, 76, 45], consistency inmotion [3, 26], viewsynthes”
Evolving Losses for Unsupervised Video Representation Learning (2020, unknown): “[12, 26, 30, 24, 31, 47, 19, 21, 45, 46, 42].”
DINOv2: Learning Robust Visual Features without Supervision (2023, background): “0% on Sketch (Wang et al., 2019) (2018) respectively – are mentioned at the top of the Table.”
Learning Temporal Dynamics from Cycles in Narrated Video (2021, background): “Another line of work uses cycles to track objects through time [20, 58], tracking a pixel forward and then backward in time and requiring that the final pixel be the same as the start pixel.”