Unsupervised Discovery of Object Landmarks as Structural Representations

By Yuting Zhang, Yijie Guo, Yixin Jin, Yijun Luo, Zhiyuan He, Honglak Lee. 2018

209 citations recorded by the source corpus.

Deep neural networks can model images with rich latent representations, but they cannot naturally conceptualize structures of object categories in a human-perceptible way. This paper addresses the problem of learning object structures in an image modeling process without supervision. We propose an autoencoding formulation to discover landmarks as explicit structural representations. The encoding module outputs landmark coordinates, whose validity is ensured by constraints that reflect the necessary properties for landmarks. The decoding module takes the landmarks as a part of the learnable input representations in an end-to-end differentiable framework. Our discovered landmarks are semantically meaningful and more predictive of manually annotated landmarks than those discovered by previous methods. The coordinates of our landmarks are also complementary features to pretrained deep-neural-network representations in recognizing visual attributes. In addition, the proposed method naturally creates an unsupervised, perceptible interface to manipulate object shapes and decode images with controllable structures.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Unsupervised Learning of Object Keypoints for Perception and Control (2019, methodology): “In recent work, object keypoint or landmark discovery methods [39, 15] have been proposed to learn representations that precisely represent locations of object parts.”
Self-supervised learning of a facial attribute embedding from video (2018, methodology): “This is achieved even though the other self-supervised methods [21, 51, 52, 64] are explicitly engineered to detect landmarks whereas our method is not.”
Unsupervised Object Keypoint Learning using Local Spatial Predictability (2020, methodology): “Recent approaches (Jakab et al., 2018; Zhang et al., 2018; Kulkarni et al., 2019; Minderer et al., 2019) apply a transformation (i.”
TAP-Vid: A Benchmark for Tracking Any Point in a Video (2022, unknown): “Keypoint discovery aims to discover keypoints and correspondence on more general objects, typically with the goal of robotic manipulation [14, 27, 28, 67, 86], but these typically train from large datasets depicting a single object.”
Unsupervised Disentanglement of Pose, Appearance and Background from Images and Videos (2020, methodology): “Our work builds upon prior methods in unsupervised discovery of image correspondences [23,25,21,22,8,7,11].”
A Review on Deep Learning Techniques for Video Prediction (2020, methodology): “Struct-VRNN [85] 2019 [209], [210] cVRNN [90], [117] RGB RGB X `2,KL SR ? ? X X”
TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement (2023, background): “Finally, some recent methods discover keypoints on moving objects [22, 23, 58, 69], though these typically require in-domain training on large datasets.”
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor With Application to In-Hand Manipulation (2020, methodology): “predictive model, based on the “Structural VRNN” architecture proposed in [31], which is also closely related to approaches proposed in [32], [33].”
Unsupervised Learning of Object Landmarks through Conditional Image Generation (2018, methodology): “[55] shares several similarities with ours, in that they also use image generation with the goal of learning landmarks.”
Animating Arbitrary Objects via Deep Motion Transfer (2018, methodology): “Inspired by recent studies on unsupervised landmark discovery for learning image representations [20, 47], we formulate the problem of learning a motion representation as an unsupervised motion-specific keypoint detection task.”
One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing (2020, methodology): “For a valid keypoint, when applying a 2D transformation to the image, the predicted keypoints should change according to the applied transformation [68, 98].”
Unsupervised Learning of Object Structure and Dynamics from Videos (2019, background): “Previous work explores learning to find keypoints in an image by applying an autoencoding architecture with keypoint-coordinates as a representational bottleneck [12, 33].”
Discovery of Latent 3D Keypoints via End-to-end Geometric Reasoning (2018, methodology): “Independent from our work, Zhang et al. [64] discover sparse 2D landmarks of images of a known object class as explicit structure representation through a reconstruction objective.”
First Order Motion Model for Image Animation (2020, methodology): “We use thin plate splines deformations as they were previously used in unsupervised keypoint detection [18, 43] and are similar to natural image deformations.”