Image2Mesh: A Learning Framework for Single Image 3D Reconstruction

By Jhony Kaesemodel Pontes, Chen Kong, Sridha Sridharan, Simon Lucey, Anders P. Eriksson, Clinton Fookes. 2017

138 citations recorded by the source corpus.

One challenge that remains open in 3D deep learning is how to efficiently represent 3D data to feed deep networks. Recent works have relied on volumetric or point cloud representations, but such approaches suffer from a number of issues such as computational complexity, unordered data, and lack of finer geometry. This paper demonstrates that a mesh representation (i.e. vertices and faces to form polygonal surfaces) is able to capture fine-grained geometry for 3D reconstruction tasks. A mesh however is also unstructured data similar to point clouds. We address this problem by proposing a learning framework to infer the parameters of a compact mesh representation rather than learning from the mesh itself. This compact representation encodes a mesh using free-form deformation and a sparse linear combination of models allowing us to reconstruct 3D meshes from single images. In contrast to prior work, we do not rely on silhouettes and landmarks to perform 3D reconstruction. We evaluate our method on synthetic and real-world datasets with very promising results. Our framework efficiently reconstructs 3D objects in a low-dimensional way while preserving its important geometrical aspects.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images (2018, neutral): “Our works are mostly related to the two recent works [17] and [24].”
Multi-View Silhouette and Depth Decomposition for High Resolution 3D Object Representation (2018, background): “A recent trend is the use of unstructured representations such as mesh models [31, 14, 44] and point clouds [33, 6] which represent the data by an unordered set with a fixed number of points.”
Differentiable surface splatting for point-based geometry processing (2019, methodology): “Numerous recent works employed DR for learning based 3D vision tasks, such as single view image reconstruction [Pontes et al. 2017; Vogels et al. 2018; Yan et al. 2016; Zhu et al. 2017], face reconstruction [Richardson et al. 2017], shape completion [Hu et al. 2019], and image synthesis [Sitzmann…”
DUSt3R: Geometric 3D Vision Made Easy (2023, unknown): “Shapes can be stored as occupancy in regular grids [19, 98, 112, 118, 138, 154, 155], octree structures [128], collections of parametric surface elements [39], point clouds encoders [29, 66, 67], free-form deformation of template meshes [89] or per-view depthmaps [54].”
Shape Prior Deformation for Categorical 6D Object Pose and Size Estimation (2020, methodology): “[13,20,41] use free-form deformation in conjunction with voxel, mesh and point cloud representations, respectively.”
Sketch2Model: View-Aware 3D Modeling from Single Free-Hand Sketches (2021, background): “With the emergence of online 3D model collections and large-scale 3D model datasets [1], data-driven approaches are developed to infer categoryspecific shapes directly from image features [3, 4, 8, 10, 16, 24, 29, 38], in the form of voxel [4, 10], point cloud [8], triangle mesh [16,29,38], and impl”
Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era (2019, methodology): “The most commonly used quantitative metrics for evaluating the accuracy of 3D reconstruction algorithms include: (1) The Mean Squared Error (MSE) [60].”