Pixel2Mesh++: Multi-View 3D Mesh Generation via Deformation

By Chao Wen, Yinda Zhang, Zhuwen Li, Yanwei Fu. 2019

285 citations recorded by the source corpus.

We study the problem of shape generation in 3D mesh representation from a few color images with known camera poses. While many previous works learn to hallucinate the shape directly from priors, we resort to further improving the shape quality by leveraging cross-view information with a graph convolutional network. Instead of building a direct mapping function from images to 3D shape, our model learns to predict series of deformations to improve a coarse shape iteratively. Inspired by traditional multiple view geometry methods, our network samples nearby area around the initial mesh's vertex locations and reasons an optimal deformation using perceptual feature statistics built from multiple input images. Extensive experiments show that our model produces accurate 3D shape that are not only visually plausible from the input perspectives, but also well aligned to arbitrary viewpoints. With the help of physically driven architecture, our model also exhibits generalization capability across different semantic categories, number of input images, and quality of mesh initialization.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

In 3D Vision & Pose

Citations in the corpus

Multiview Compressive Coding for 3D Reconstruction (2023, methodology): “Supervised approaches predict voxels [67, 74] or meshes [69, 71] by training deep nets.”
One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization (2023, methodology): “By combing 2D image encoder and 3D generators, they generates 3D data in various representations, including 3D voxels [18, 79, 10, 81, 80, 85], point clouds [15, 88, 19, 1, 44, 90], polygon meshes [30, 73, 77, 51], and parametric models [54, 94, 95].”
Shape Prior Deformation for Categorical 6D Object Pose and Size Estimation (2020, background): “[34,36] starts from a coarse shape and predicts a series of deformations to progressively improve the”
Convolutional Occupancy Networks (2020, background): “Meshes: A popular alternative is to directly regress the vertices and faces of a mesh [12, 13, 17, 20, 22, 44, 45] using a neural network.”
NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction (2021, background): “These inductive biases can be explicit representations, such as voxel grids [13, 5, 44], point cloud [7, 24, 18], meshes [41, 43, 14], and implicit representations.”
DIST: Rendering Deep Implicit Signed Distance Function With Differentiable Sphere Tracing (2019, background): “Deep learning based multi-view shape prediction methods [53, 15, 16, 49, 50] further involve geometric constraints across views in the deep learning framework, which shows better generalization.”
Multi-View Mesh Reconstruction with Neural Deferred Shading (2022, unknown): “Recently, learned image features and neural shape priors have been used to drive the mesh deformation process [30,62].”