Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images
By Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei David Liu, Yu-Gang Jiang. 2018
1,502 citations recorded by the source corpus.
We propose an end-to-end deep learning architecture that produces a 3D shape in triangular mesh from a single color image. Limited by the nature of deep neural network, previous methods usually represent a 3D shape in volume or point cloud, and it is non-trivial to convert them to the more ready-to-use mesh model. Unlike the existing methods, our network represents 3D mesh in a graph-based convolutional neural network and produces correct geometry by progressively deforming an ellipsoid, leveraging perceptual features extracted from the input image. We adopt a coarse-to-fine strategy to make the whole deformation procedure stable, and define various of mesh related losses to capture properties of different levels to guarantee visually appealing and physically accurate 3D geometry. Extensive experiments show that our method not only qualitatively produces mesh model with better details, but also achieves higher 3D shape estimation accuracy compared to the state-of-the-art.
Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models, 3D Vision & Pose
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computational Mechanics
- Andreas Geiger
- Gerard Pons-Moll
- Michael Niemeyer
- Michael J. Black
- Jitendra Malik
- Matthias Nießner
- Angjoo Kanazawa
- Justin Johnson
In Computer Graphics and Computer-Aided Design
- Andreas Geiger
- Jonathan T. Barron
- Michael Niemeyer
- Matthias Nießner
- Justin Johnson
- Songyou Peng
- Matthew Tancik
- Angjoo Kanazawa
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Jitendra Malik
- Ming-Hsuan Yang
- Trevor Darrell
- Xiaogang Wang
- Andrea Vedaldi
- Andrew Zisserman
In Generative Models
- Andreas Geiger
- Jun-Yan Zhu
- Daniel Cohen-Or
- Eli Shechtman
- Richard Zhang
- Ming-Hsuan Yang
- Alexei A. Efros
- Jitendra Malik
In 3D Vision & Pose
- Andreas Geiger
- Michael Niemeyer
- Jitendra Malik
- Gerard Pons-Moll
- Justin Johnson
- Matthias Nießner
- Songyou Peng
- Michael J. Black
Citations in the corpus
- Multiview Compressive Coding for 3D Reconstruction (2023, background): “Others [16,68] predict 3D from a single image but rely on expensive CAD supervision [5, 59].”
- Occupancy Networks: Learning 3D Reconstruction in Function Space (2018, methodology): “Existing representations can be broadly categorized into three categories: voxel-based representations [4,19,43,58,64,69,75] , point-based representations [1,17] and mesh representations [34,57,70], see Fig.”
- Learning Implicit Fields for Generative Shape Modeling (2018, methodology): “There have been a variety of 3D shape representations for deep learning of shapes, such as voxel grids [9, 14, 28, 45, 46], octrees [18, 35, 40, 43, 44], multi-view images [29, 39], point clouds [1, 12, 13, 32, 33, 47, 48], geometry images [37, 38], deformable mesh/patches [16, 38, 42, 47], and part”
- Multi-View Silhouette and Depth Decomposition for High Resolution 3D Object Representation (2018, methodology): “Our surface accuracy results can be seen in table 3 compared to [44, 6, 14, 3].”
- PointRend: Image Segmentation As Rendering (2019, background): “Instead, recent works consider more efficient non-uniform representations such as meshes [42, 13], signed distance functions [32], and oc-trees [41].”
- Few-Shot Generalization for Single-Image 3D Reconstruction via Priors (2019, methodology): “Closer in spirit to our work are single-view reconstruction methods that use meshes as their underlying representation, which often function by deforming a prior mesh [23, 9] However, in all these approaches, the focus is on improving the in-category performance rather than on generalization or tran”
- Mesh R-CNN (2019, methodology): “Similar to [69], each re-finement stage consists of three operations: vertex alignment , which extracts image features for vertices; graph convolution , which propagates information along mesh edges; and vertex refinement , which updates vertex positions.”
- Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis (2021, neutral): “Compared to methods that learn to directly generate explicit representations, such as meshes [54], by committing to a preset topology, our DMT ET can produce shapes with arbitrary topology.”
- What's Behind the Couch? Directed Ray Distance Functions (DRDF) for 3D Scene Reconstruction (2021, neutral): “Voxel-based methods [10, 20] scale poorly with size due to their memory requirements, and mesh-based ones [64] Project Page https : / / nileshkulkarni . github . io / scene_drdf struggle with varying topology.”
- Unsupervised Learning of 3D Object Categories from Videos in the Wild (2021, background): “[44, 1] label each 3D point with its signed distance to the nearest surface point, [35, 6] predict binary per-point occupancies, [11, 10] proposed more structured occupancy functions, and [13, 60] reconstruct meshes from single views.”
- Pix2Vox: Context-Aware 3D Reconstruction From Single and Multi-View Images (2019, methodology): “Consequently, these methods cannot recover 3D volumes accurately [24].”
- Deep Stereo Using Adaptive Thin Volume Representation With Uncertainty Awareness (2019, background): “Recent work also tried to directly reconstruct surface meshes [34, 25, 53, 19, 46, 28], deformable shapes [24, 25], and some learned implicit distance functions [7, 41, 37, 6].”
- Point-E: A System for Generating 3D Point Clouds from Complex Prompts (2022, methodology): “Nevertheless, some regression-based methods have shown promising results on this task (Choy et al., 2016; Wang et al., 2018; Gkioxari et al., 2019; Groueix et al., 2018; Yu et al., 2020; Lin et al., 2022b).”
- MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion (2024, methodology): “Commonly used output representations include point clouds (Guo et al., 2020; Lin et al., 2018), meshes (Gkioxari et al., 2019; Wang et al., 2018), voxel (Sitzmann et al., 2019; Choy et al., 2016; Tulsiani et al., 2017), implicit representation (Wang et al., 2021a; Peng et al., 2020; Chen & Zhang,…”
- PyMAF: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loop (2021, background): “For generic objects, Pixel2Mesh [56] progressively deforms an initial ellipsoid by leveraging perceptual features.”
- HyperCube: Implicit Field Representations of Voxelized 3D Models (2021, background): “3D objects can be represented using different approaches including voxel grids [4, 5, 6, 7], octrees [8, 9, 10, 11], multi-view images [12, 13, 14], point clouds [15, 16, 17, 18, 19], geometry images [20, 21], deformable meshes [5, 21, 22, 19], and part-based structural graphs [23, 24].”
- GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis (2020, background): “Key advantages over voxel [6,9,50,52, 65,66] or mesh-based [15,26,43,62] methods are that they do not discretize space and are not restriced in topology.”
- Self-supervised Learning of 3D Objects from Natural Images (2019, methodology): “One research direction is how neural networks handle irregular 3D representations, such as high-resolution voxels [11, 37], point clouds [7], meshes [9, 43] and implicit functions [3, 24, 26].”
- AutoSDF: Shape Priors for 3D Completion, Reconstruction and Generation (2022, neutral): “Several approaches have shown impressive single-view reconstruction results using voxels [10, 14, 31, 38, 39], point clouds [13, 19, 41], meshes [36, 37], and most recently implicit representations of 3D surfaces like SDFs [17,43], UDFs [9] and CSPs [35].”
- PGSR: Planar-Based Gaussian Splatting for Efficient and High-Fidelity Surface Reconstruction (2024, positive): “Numerous pioneering efforts have leveraged pure deep neural networks to predict surface models directly from single or multiple image conditions using point clouds [43], [44], voxels [45], [46], and triangular meshes [7], [47] or implicit fields [48], [49] in end-to-end manner.”