Learning Dense Correspondence via 3D-Guided Cycle Consistency

By Tinghui Zhou, Philipp Krahenbuhl, Mathieu Aubry, Qi-Xing Huang, Alexei A. Efros. 2016

405 citations recorded by the source corpus.

Discriminative deep learning approaches have shown impressive results for problems where human-labeled ground truth is plentiful, but what about tasks where labels are difficult or impossible to obtain? This paper tackles one such problem: establishing dense visual correspondence across different object instances. For this task, although we do not know what the ground-truth is, we know it should be consistent across instances of that category. We exploit this consistency as a supervisory signal to train a convolutional neural network to predict cross-instance correspondences between pairs of images depicting objects of the same category. For each pair of training images we find an appropriate 3D CAD model and render two synthetic views to link in with the pair, establishing a correspondence flow 4-cycle. We use ground-truth synthetic-to-synthetic correspondences, provided by the rendering engine, to train a ConvNet to predict synthetic-to-real, real-to-real and real-to-synthetic correspondences that are cycle-consistent with the ground-truth. At test time, no CAD models are required. We demonstrate that our end-to-end trained ConvNet supervised by cycle-consistency outperforms state-of-the-art pairwise matching methods in correspondence-related tasks.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

DensePose: Dense Human Pose Estimation in the Wild (2018, background): “Several other works have recently aimed at recovering dense correspondences between pairs [3] or sets of RGB images [50, 10] in an unsupervised setting.”
Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks (2017, methodology): “More recently, higher-order cycle consistency has been used in structure from motion [61], 3D shape matching [21], cosegmentation [55], dense semantic alignment [65, 64], and depth estimation [14].”
Single Image 3D Interpreter Network (2016, methodology): “[59] also used a similar strategy for object detection and matching, respectively.”
Playing hard exploration games by watching YouTube (2018, positive): “Motivated by the success of cyclic relations in CycleGAN [48] and for matching visual features across images [47], we propose cycle-consistency for this purpose.”
Tracking Emerges by Colorizing Videos (2018, methodology): “Moreover, our method is trained directly on natural data without the use of computer generated graphics [22,23].”
Zero-Shot Visual Imitation (2018, background): “Enforcing consistency as a meta-supervision has also been successful in finding visual correspondences (Zhou et al., 2016) or unpaired image translations (Zhu et al.”
Disentangling Factors of Variation with Cycle-Consistent Variational Auto-Encoders (2018, background): “Zhou et al. [11] use cycle-consistency to establish cross-instance correspondences between pairs of images depicting objects of the same category.”
StarMap for Category-Agnostic Keypoint and Viewpoint Estimation (2018, background): “To merge keypoints from diﬀerent objects, one has to establish consistent correspondences [50] between diﬀerent keypoints across multiple categories, which is diﬃcult or sometimes impossible.”
Few-Shot Segmentation via Cycle-Consistent Transformer (2021, background): “It is also effective in learning 3D correspondence [42], consistency between video frames [33] and association between different domains [14].”
Recycle-GAN: Unsupervised Video Retargeting (2018, background): “Cycle loss: Zhu et al. [53] use cycle consistency [51] to deﬁne a reconstruction loss when the pairs are not available.”
Unsupervised Monocular Depth Estimation with Left-Right Consistency (2016, methodology): “[41, 54, 55], our model overcomes this problem by using bilinear sampling [26] to generate images, resulting in a fully (sub-)differentiable training loss.”
3D Shape Segmentation with Projective Convolutional Networks (2016, background): “Compositional part-based reasoning for 3D shapes has been shown to be effective for a large number of vision, robotics and virtual reality applications, such as cross-modal analysis of 3D shapes and color images [60, 24], skeletal tracking [42], objection detection in images [11, 30, 36], 3D object ”
Real-Time Monocular Depth Estimation Using Synthetic Data with Domain Adaptation via Image Style Transfer (2018, methodology): “This approach uses adversarial training [27] and cycle-consistency [26, 79, 89, 86] to translate between two sets of unaligned images from different domains.”
Taming Visually Guided Sound Generation (2021, background): “[24] improved the performance of the GAN with cross-modal cycle-consistency [72].”
Lifting AutoEncoders: Unsupervised Learning of a Fully-Disentangled 3D Morphable Model Using Deep Non-Rigid Structure From Motion (2019, background): “Over the past few years rapid progress has been made in the direction of disentangling the latent space of deep models into dimensions that account for generic factors of variation, such as identity and low-dimensional transformations [8, 52, 28, 51, 39], or even non-rigid, dense deformations from a”
Progressive Transfer Learning and Adversarial Domain Adaptation for Cross-Domain Skin Disease Classification (2020, background): “In addition, it introduces a cycle loss [43] to keep cycle consistency, so that for any image sample x ∈ X S , the generated sample G ST ( x ) that is mapped to the target domain X T can be converted back to the source domain, i.e.”
Neural best-buddies (2018, methodology): “2017], 3D Cycle Consistency [Zhou et al. 2016], and Deep Image”