Discovery of Latent 3D Keypoints via End-to-end Geometric Reasoning

By Supasorn Suwajanakorn, Noah Snavely, Jonathan Tompson, Mohammad Norouzi. 2018

261 citations recorded by the source corpus.

This paper presents KeypointNet, an end-to-end geometric reasoning framework to learn an optimal set of category-specific 3D keypoints, along with their detectors. Given a single image, KeypointNet extracts 3D keypoints that are optimized for a downstream task. We demonstrate this framework on 3D pose estimation by proposing a differentiable objective that seeks the optimal set of keypoints for recovering the relative pose between two views of an object. Our model discovers geometrically and semantically consistent keypoints across viewing angles and instances of an object category. Importantly, we find that our end-to-end framework using no ground-truth keypoint annotations outperforms a fully supervised baseline using the same neural network architecture on the task of pose estimation. The discovered 3D keypoints on the car, chair, and plane categories of ShapeNet [6] are visualized at keypointnet.github.io.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Unsupervised Learning of Object Keypoints for Perception and Control (2019, background): “Other works similarly either require known transformations or output dense correspondences instead of discrete landmarks [35, 29, 33, 37].”
Deep Dynamics Models for Learning Dexterous Manipulation (2019, methodology): “We employ a dilated CNN model for object tracking, modeled after KeypointNet [36], which takes a 280x180 RGB stereo image pair as input from a calibrated 12 cm baseline camera rig.”
Neuroevolution of self-interpretable agents (2020, result): “Conceptually, our approach is similar to Spatial Softmax [26, 57, 96],”
Key.Net: Keypoint Detection by Handcrafted and Learned CNN Filters (2019, methodology): “Extracting coordinates for training keypoint detectors has been widely studied and showed great improvements: [7, 9, 10] extracted coordinates in a patch level, SuperPoint [13] used a channel-wise softmax to get maxima belonging to fix grids of 8x8, and [35] used a spatial softmax layer to compute t”
Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objects (2018, background): “For alternate approaches to related problems, see [33, 34, 35, 36, 37].”
KeyPose: Multi-View 3D Labeling and Keypoint Estimation for Transparent Objects (2019, methodology): “A more flexible method uses 3D keypoints [16, 26], which can handle articulated and deformable ob-”
Unsupervised Learning of Object Landmarks through Conditional Image Generation (2018, background): “[43] learn 3D-keypoints for objects from two images which differ by a known 3D transformation, by enforcing equivariance [45].”
PVN3D: A Deep Point-Wise 3D Keypoints Voting Network for 6DoF Pose Estimation (2019, background): “[45] extracts 3D keypoints from two views of synthetic RGB images to recover 3D poses.”
PRNet: Self-Supervised Learning for Partial-to-Partial Registration (2019, methodology): “Inspired by this observation, self-supervised learning usually involves predicting part of an input from another part [47, 48], solving one task using features learned from another task [45] and/or enforcing consistency from different views/modalities [49, 50].”
Unsupervised part representation by Flow Capsules (2020, background): “KeypointNet (Suwajanakorn et al., 2019) addresses the problem of keypoint discovery.”
Unsupervised Learning of Probably Symmetric Deformable 3D Objects From Images in the Wild (2019, methodology): “Other approaches have used single views with 2D keypoint annotations [27, 37, 51, 6] or object masks [27, 7].”
Few-Shot Object Detection and Viewpoint Estimation for Objects in the Wild (2020, unknown): “To overcome this limitation, recent works propose to conduct unsupervised viewpoint estimation [66] or predict generic 3D keypoints for all object classes [8].”
iNeRF: Inverting Neural Radiance Fields for Pose Estimation (2020, unknown): “oints with known 3D models [1,5,7,33]. Recent approaches based on deep learning have proposed to 1) directly estimate objects pose using CNN-based architectures [36,44,50] or 2) estimate 2D keypoints [32, 40,42,43] and solve for pose using the PnP-RANSAC algorithm. Differentiable mesh renderers [3,2”
Self-Supervised Learning of 3D Human Pose Using Multi-View Geometry (2019, methodology): “In fact, other multi-view methods faced the same problem [31, 37].”
Self-supervised Learning of 3D Objects from Natural Images (2019, background): “Incorporating unsupervised learning of keypoints [36] would alleviate this problem.”
DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion (2019, methodology): “Newer methods address the challenge by learning to predict the 2D keypoints [3, 22, 32, 34, 35] and solve the poses by PnP [10].”
Repurposing GANs for One-Shot Semantic Part Segmentation (2021, background): “been reported on objects with symmetry in an unsupervised method [70] (face images are an exception because they are mostly frontal and heavily canonicalized), and one may need an additional orientation signal [70] or more training examples.”