Learning Category-Specific Mesh Reconstruction from Image Collections

By Angjoo Kanazawa, Shubham Tulsiani, Alexei A. Efros, Jitendra Malik. 2018

662 citations recorded by the source corpus.

We present a learning framework for recovering the 3D shape, camera, and texture of an object from a single image. The shape is represented as a deformable 3D mesh model of an object category where a shape is parameterized by a learned mean shape and per-instance predicted deformation. Our approach allows leveraging an annotated image collection for training, where the deformable model and the 3D prediction mechanism are learned without relying on ground-truth 3D or multi-view supervision. Our representation enables us to go beyond existing 3D prediction approaches by incorporating texture inference as prediction of an image in a canonical appearance space. Additionally, we show that semantic keypoints can be easily associated with the predicted shapes. We present qualitative and quantitative results of our approach on the CUB dataset, and show that we can learn to predict the diverse shapes and textures across birds using only an annotated image collection. We also demonstrate the the applicability of our method for learning the 3D structure of other generic categories.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

Occupancy Networks: Learning 3D Reconstruction in Function Space (2018, background): “Point clouds [1,17] and meshes [34,57,70] have been introduced as alternative representations for deep learning, using appropriate loss functions.”
HoloGAN: Unsupervised Learning of 3D Representations From Natural Images (2019, background): “HoloGAN can be trained end-to-end in an unsupervised manner using only unlabelled 2D images, without any supervision of poses, 3D shapes, multiple views of objects, or geometry priors such as symmetry and smoothness over the 3D representation that are common in this line of work [3, 26, 45].”
Learning Joint Reconstruction of Hands and Manipulated Objects (2019, positive): “Following [14, 22, 73] we employ two losses that penalize irregular meshes.”
One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization (2023, methodology): “To endow intelligent agents with this ability, many existing methods [30, 18, 24, 10, 81, 85, 15, 77] exploit class-speciﬁc priors by training 3D generative networks on 3D shape datasets [4].”
PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization (2019, methodology): “While texture inference is often performed on either a 2D parameterization of the surface [31, 21] or in view-space [39], PIFu enables us to directly predict the RGB colors on the surface geometry by deﬁning s in Eq.”
Soft Rasterizer: Differentiable Rendering for Unsupervised Single-View Mesh Reconstruction (2019, background): “[17] strive to reconstruct category-specific mesh from image collections”
Soft Rasterizer: A Differentiable Renderer for Image-Based 3D Reasoning (2019, methodology): “To simplify the learning problem, recent works reconstruct 3D shape via predicting intermediate 2.5D representations, such as depth map [25], image collections [18], displacement map [16] or normal map [36, 44].”
Pixel2Mesh++: Multi-View 3D Mesh Generation via Deformation (2019, background): “[19] learn a category-speciﬁc 3D deformable model and reasons the shape deformations in different images.”
Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer (2019, methodology): “Results: We compare our method with CMR [13].”
SynSin: End-to-End View Synthesis From a Single Image (2019, background): “Many works explore using a DNN to predict 3D object shapes [18, 20, 24, 26, 64, 69] or the depth of a scene given an image [6, 13, 33, 73].”
Neighbourhood Consensus Networks (2018, background): “These results open up the possibility for end-to-end learning of other challenging visual correspondence tasks, such as 3D category-level matching [18], or visual localization across day/night illumination [31].”
Do 2D GANs Know 3D Shape? Unsupervised 3D shape reconstruction from 2D Image GANs (2020, background): “And Tran & Liu (2018); Kanazawa et al. (2018b) adopt the guidance of 2D key-points to learn the 3D shape of face or objects.”
Deep Stereo Using Adaptive Thin Volume Representation With Uncertainty Awareness (2019, background): “Recent work also tried to directly reconstruct surface meshes [34, 25, 53, 19, 46, 28], deformable shapes [24, 25], and some learned implicit distance functions [7, 41, 37, 6].”
Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations (2019, methodology): “Discretization-based techniques use voxel grids [7, 16–22], octree hierarchies [23–25], point clouds [11, 26, 27], multiplane images [28], patches [29], or meshes [15, 21, 30, 31].”
Articulation-Aware Canonical Surface Mapping (2020, methodology): “Similarly, approaches pursuing articulation inference [16, 43] also rely on keypoint annotation to enable”
Mesh R-CNN (2019, methodology): “However, prior work on mesh prediction with deep networks [23, 57, 69] has been constrained to deform from”