Unsupervised Learning of Object Landmarks by Factorized Spatial Embeddings

By James Thewlis, Hakan Bilen, Andrea Vedaldi. 2017

171 citations recorded by the source corpus.

Learning automatically the structure of object categories remains an important open problem in computer vision. In this paper, we propose a novel unsupervised approach that can discover and learn landmarks in object categories, thus characterizing their structure. Our approach is based on factorizing image deformations, as induced by a viewpoint change or an object deformation, by learning a deep neural network that detects landmarks consistently with such visual effects. Furthermore, we show that the learned landmarks establish meaningful correspondences between different object instances in a category without having to impose this requirement explicitly. We assess the method qualitatively on a variety of object types, natural and man-made. We also show that our unsupervised landmarks are highly predictive of manually-annotated landmarks in face benchmark datasets, and can be used to regress these with a high degree of accuracy.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Unsupervised Learning of Object Keypoints for Perception and Control (2019, background): “Other works similarly either require known transformations or output dense correspondences instead of discrete landmarks [35, 29, 33, 37].”
Lifting AutoEncoders: Unsupervised Learning of a Fully-Disentangled 3D Morphable Model Using Deep Non-Rigid Structure From Motion (2019, methodology): “Similarly to [42, 41, 36], we evaluate the extent to which landmarks are captured by our 3D shape model by training a linear regressor to predict them given the locations of the mesh vertices in 3D.”
Repurposing GANs for One-Shot Semantic Part Segmentation (2021, methodology): “unsupervisedly with no human references, but they can be evaluated under the few-shot context, as commonly done in [66], [67], [68], by regressing the predicted locations to human-labeled locations with a linear regression.”
Self-supervised learning of a facial attribute embedding from video (2018, methodology): “To provide further supervision from image data, the images themselves can be transformed via a synthetic warp or rotation and the network trained to recognise the rotation [19] or to learn equivariant pixel embeddings [40, 51, 52].”
Self-Supervised Learning of Geometrically Stable Features Through Probabilistic Introspection (2018, methodology): “[32] show that placing constraints on matching builds object landmarks that are not only consistently detected across the deformation of a current instance, but also across instances.”
TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement (2023, background): “Finally, some recent methods discover keypoints on moving objects [22, 23, 58, 69], though these typically require in-domain training on large datasets.”
Integral Human Pose Regression (2017, methodology): “It has been known as soft-argmax and used in the previous works [27,52,45].”
TAP-Vid: A Benchmark for Tracking Any Point in a Video (2022, unknown): “Keypoint discovery aims to discover keypoints and correspondence on more general objects, typically with the goal of robotic manipulation [14, 27, 28, 67, 86], but these typically train from large datasets depicting a single object.”
Unsupervised Disentanglement of Pose, Appearance and Background from Images and Videos (2020, methodology): “As with prior works [7,22], we fit a linear regressor (without intercept) to our learned landmark locations from our pose representation to supervised keypoint coordinates.”
Unsupervised object learning from dense equivariant image labelling (2017, methodology): “Second, we show that the equivariance constraint proposed in [30] is not quite enough to learn such an embedding; it must be complemented with the concept of a distinctive embedding (section 3.”