C3DPO: Canonical 3D Pose Networks for Non-Rigid Structure From Motion

By David Novotný, Nikhila Ravi, Benjamin Graham, Natalia Neverova, Andrea Vedaldi. 2019

129 citations recorded by the source corpus.

We propose C3DPO, a method for extracting 3D models of deformable objects from 2D keypoint annotations in unconstrained images. We do so by learning a deep network that reconstructs a 3D object from a single view at a time, accounting for partial occlusions, and explicitly factoring the effects of viewpoint changes and object deformations. In order to achieve this factorization, we introduce a novel regularization technique. We first show that the factorization is successful if, and only if, there exists a certain canonicalization function of the reconstructed shapes. Then, we learn the canonicalization function together with the reconstruction one, which constrains the result to be consistent. We demonstrate state-of-the-art reconstruction results for methods that do not use ground-truth 3D supervision for a number of benchmarks, including Up3D and PASCAL3D+.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

Multiview Compressive Coding for 3D Reconstruction (2023, methodology): “Reminiscent of generalized cylinders [41], some introduce object-specific priors via category-specific 3D templates [26,30,32], pose [43] or symmetries [73].”
Canonical Capsules: Unsupervised Capsules in Canonical Pose (2020, background): “C3DPO [34] learns a canonical 3D frame based on 2D input keypoints.”
PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Tracking (2023, background): “Early motion estimation methods cast point tracking as an optimization problem defined on handcrafted features [27, 37, 57, 52, 12], and these techniques continue to drive structure-from-motion [11, 31, 43] and simultaneous localization and mapping systems [54].”
Deep 3D human pose estimation: A review (2021, unknown): “Similarly, Novotny et al. (2019) propose to learn a canonicalization network that maps equivalent class members to the canonical reconstruction to regularize the results.”
Unsupervised Learning of Probably Symmetric Deformable 3D Objects From Images in the Wild (2019, background): “Second, Non-Rigid SfM (NRSfM) approaches [4, 41] can learn to reconstruct deformable objects by allowing 3D points to deform in a limited manner between views, but require supervision in terms of annotated 2D keypoints for both training and testing.”
LRM: Large Reconstruction Model for Single Image to 3D (2023, unknown): “Leveraging 3D templates (Roth et al., 2016; Goel et al., 2020; Kanazawa et al., 2018; Kulkarni et al., 2020), semantics (Li et al., 2020), and poses (Bogo et al., 2016; Novotny et al., 2019) as shape priors have also been widely studied in category-specific reconstruction.”
Pix2Shape: Towards Unsupervised Learning of 3D Scenes from Images Using a View-Based Representation (2020, background): “…such asmultiple views of the same object (Yan et al. 2016; Tulsiani et al. 2017; Li et al. 2019), 2.5D supervision (Wu et al. 2016a, 2017), key-point (Kar et al. 2014; Novotný et al. 2019) and silhouette annotations (Wiles and Zisserman 2017; Henderson and Ferrari 2018; Chen et al. 2019).”
Accelerating 3D deep learning with PyTorch3D (2019, background): “Predicting depth [10, 9], or 3D shape [8, 11, 59, 41] can lift 2D images into 3D. Generative models [61, 62, 39] might one day aid artists in authoring 3D content.”