Unsupervised Learning of Object Landmarks through Conditional Image Generation

By Tomas Jakab, Ankush Gupta, Hakan Bilen, Andrea Vedaldi. 2018

271 citations recorded by the source corpus.

We propose a method for learning landmark detectors for visual objects (such as the eyes and the nose in a face) without any manual supervision. We cast this as the problem of generating images that combine the appearance of the object as seen in a first example image with the geometry of the object as seen in a second example image, where the two examples differ by a viewpoint change and/or an object deformation. In order to factorize appearance and geometry, we introduce a tight bottleneck in the geometry-extraction process that selects and distils geometry-related features. Compared to standard image generation problems, which often use generative adversarial networks, our generation task is conditioned on both appearance and geometry and thus is significantly less ambiguous, to the point that adopting a simple perceptual loss formulation is sufficient. We demonstrate that our approach can learn object landmarks from synthetic image deformations or videos, all without manual supervision, while outperforming state-of-the-art unsupervised landmark detectors. We further show that our method is applicable to a large variety of datasets - faces, people, 3D objects, and digits - without any modifications.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

On the Binding Problem in Artificial Neural Networks (2020, background): “Recently, there has been renewed interest in saliency-based approaches through the discovery of keypoints (Jakab et al., 2018; Kulkarni et al., 2019; Minderer et al., 2019; Gopalakrishnan et al., 2020).”
Unsupervised Learning of Object Keypoints for Perception and Control (2019, methodology): “We follow the formulation of [15] and assume access to frame pairs xs,xt collected from some trajectories such that the frames differ only in objects’ pose / geometry or appearance.”
TAP-Vid: A Benchmark for Tracking Any Point in a Video (2022, unknown): “Keypoint discovery aims to discover keypoints and correspondence on more general objects, typically with the goal of robotic manipulation [14, 27, 28, 67, 86], but these typically train from large datasets depicting a single object.”
First Order Motion Model for Image Animation (2020, methodology): “Third, we extend the equivariance loss commonly used for keypoints detector training [18, 44], to improve the estimation of local affine transformations.”
Unsupervised Disentanglement of Pose, Appearance and Background from Images and Videos (2020, methodology): “Our work builds upon existing methods in image-reconstruction-guided landmark learning techniques [7,11].”
TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement (2023, background): “Finally, some recent methods discover keypoints on moving objects [26, 27, 68, 79], though these typically require in-domain training on large datasets.”
Deep ViT Features as Dense Visual Descriptors (2021, methodology): “First three methods, listed for reference, are specially designed for landmarks discovery, IMM [21] specializes on faces.”
Generative Hierarchical Features from Synthesizing Images (2020, background): “We compare our approach with some supervised and unsupervised alternatives [63, 59, 28, 19].”
Act the Part: Learning Interaction Strategies for Articulated Object Part Discovery (2021, methodology): “Since we wish to infer pushing based on holding, we encode the hold as a 2D Gaussian h t centered at the hold location with standard deviation of one pixel [20].”
Sim2real transfer learning for 3D human pose estimation: motion to the rescue (2019, background): “2D keypoint detection is far easier to annotate than 3D, and even when 2D keypoints are not available, some recent works have argued that 2D correspondence and keypoints can even be obtained in a self-supervised manner [30, 95].”
Lifting AutoEncoders: Unsupervised Learning of a Fully-Disentangled 3D Morphable Model Using Deep Non-Rigid Structure From Motion (2019, unknown): “(2018) [28] 2.”
Unsupervised Object Keypoint Learning using Local Spatial Predictability (2020, methodology): “2D co-ordinates are computed from these K maps as described in Jakab et al. (2018).”
Repurposing GANs for One-Shot Semantic Part Segmentation (2021, methodology): “unsupervisedly with no human references, but they can be evaluated under the few-shot context, as commonly done in [66], [67], [68], by regressing the predicted locations to human-labeled locations with a linear regression.”
Animating Arbitrary Objects via Deep Motion Transfer (2018, methodology): “Inspired by recent studies on unsupervised landmark discovery for learning image representations [20, 47], we formulate the problem of learning a motion representation as an unsupervised motion-specific keypoint detection task.”