MarrNet: 3D Shape Reconstruction via 2.5D Sketches

By Jiajun Wu, Tianfan Xue, Xingyuan Sun, Bill Freeman, Josh Tenenbaum. 2017

440 citations recorded by the source corpus.

3D object reconstruction from a single image is a highly under-determined problem, requiring strong prior knowledge of plausible 3D shapes. This introduces challenges for learning-based approaches, as 3D object annotations are scarce in real images. Previous work chose to train on synthetic data with ground truth 3D information, but suffered from domain adaptation when tested on real data. In this work, we propose MarrNet, an end-to-end trainable model that sequentially estimates 2.5D sketches and 3D object shape. Our disentangled, two-step formulation has three advantages. First, compared to full 3D shape, 2.5D sketches are much easier to be recovered from a 2D image; models that recover 2.5D sketches are also more likely to transfer from synthetic to real data. Second, for 3D reconstruction from 2.5D sketches, systems can learn purely from synthetic data. This is because we can easily render realistic 2.5D sketches without modeling object appearance variations in real images, including lighting, texture, etc. This further relieves the domain adaptation problem. Third, we derive differentiable projective functions from 3D shape to 2.5D sketches; the framework is therefore end-to-end trainable on real images, requiring no human annotations. Our model achieves state-of-the-art performance on 3D shape reconstruction.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

Multiview Compressive Coding for 3D Reconstruction (2023, background): “One line of work trains models that predict 3D geometry via CAD [16, 68], meshes [30,75], voxels [15,72] or point clouds [12,36] supervision.”
Occupancy Networks: Learning 3D Reconstruction in Function Space (2018, background): “While recent works [74, 76, 79] have applied 3D convolutional neural networks to resolutions up to 128 3 , this is only possible with shallow architectures and small batch sizes, which leads to slow training.”
Learning to Reconstruct Shapes from Unseen Classes (2018, methodology): “Building upon the MarrNet framework [Wu et al., 2017], we further decompose f2.”
Learning Shape Priors for Single-View 3D Completion and Reconstruction (2018, methodology): “With the development of large-scale shape repositories like ShapeNet [5] and methods like deep convolutional networks, researchers have built more scalable and eﬃcient models in recent years [7,13,18,27,36,38,48,52,56,57,62].”
Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling (2018, methodology): “Our model is an extension of MarrNet [60], both of which use 2.”
Learning Category-Specific Mesh Reconstruction from Image Collections (2018, methodology): “Relying on ground-truth 3D supervision (using synthetic data), some approaches have examined learning voxel [4,8,39,33], point cloud [7] or octree [10,26] prediction.”
Multi-View Silhouette and Depth Decomposition for High Resolution 3D Object Representation (2018, neutral): “MarrNet [47], which resembles our work, models 3D objects through the use of 2.”
3D-FUTURE: 3D Furniture Shape with TextURE (2020, background): “Recent works (Sun et al., 2018a; Huang et al., 2018; Wu et al., 2017; Bansal et al., 2016; Bachman, 1978; Grabner et al., 2018, 2019) predict 2.5D sketches from images, such as surface normal, depth, and location ﬁeld, to bridge the gaps between 3D and 2D domains.”
Neural Unsigned Distance Fields for Implicit Function Learning (2020, background): “Higher resolutions (256 3 ) [82, 81, 87] can be achieved at the cost of slow training, or diﬃcult multi-resolution implementations [31, 72, 77].”
BEHAVE: Dataset and Method for Tracking Human Object Interactions (2022, background): “Object reconstruction Most existing work on reconstructing 3D objects from RGB [21, 46, 53, 75, 78] and RGBD [45, 55, 82] data does so in isolation, without the human involvement or the interaction.”
Zero-1-to-3: Zero-shot One Image to 3D Object (2023, background): “One line of work builds priors from relying on collections of 3D primitives represented as meshes [58, 62], voxels [16, 60], or point clouds [12, 30], and use image encoders for conditioning.”
3D-Aware Scene Manipulation via Inverse Graphics (2018, neutral): “Many recent papers have explored the problem of 3D reconstruction from a single color image, depth map, or silhouette [Choy et al., 2016, Kar et al., 2015, Tatarchenko et al., 2016, Tulsiani et al., 2017, Wu et al., 2017b, 2016b, Yan et al., 2016b, Soltani et al., 2017].”
Learning to Infer and Execute 3D Shape Programs (2019, methodology): “We separately train a MarrNet (Wu et al., 2017) model for chairs and tables on ShapeNet, and then reconstruct 3D shapes from 2D images on the Pix3D dataset.”
StarMap for Category-Agnostic Keypoint and Viewpoint Estimation (2018, background): “Accurate semantic keypoint detection forms the basis for many visual understanding tasks, including human pose estimation [4,24,27,53], hand pose estimation [48,54], viewpoint estimation [26,37], feature matching [16], ﬁne-grained image classiﬁcation [49], and 3D reconstruction [38,41,11,10].”
3D Object Dense Reconstruction from a Single Depth View (2018, methodology): “[61] designed a pseudo-renderer to predict dense 3D shapes, while [62] sequentially estimates 2.”
Lifting AutoEncoders: Unsupervised Learning of a Fully-Disentangled 3D Morphable Model Using Deep Non-Rigid Structure From Motion (2019, background): “on [47], self supervision [29] or multiview supervision [82]. These works can be understood in D. Marr’s terms as getting 2.5D proxies to 3D geometry, which could eventually lead to 3D reconstruction [80]: texture is determined 0 0 0 1 ........ .... V render er Learned Base Shape Input I mage Predic”
Visual Object Networks: Image Generation with Disentangled 3D Representations (2018, methodology): “Given a synthesized voxelized shape v = Gshape(zshape), how can we connect it to a 2D image? Inspired by recent work on 3D reconstruction [Wu et al., 2017], we use 2.”
Image2Mesh: A Learning Framework for Single Image 3D Reconstruction (2017, background): “Several papers have proposed a 3D volumetric representation [3,4,10,11,20,24,27,30,33,35,38] so they can feed deep neural networks applying 3D convolutions, pooling, and other techniques that have been successfully applied to 2D images for the learning process.”
Pix2Vox: Context-Aware 3D Reconstruction From Single and Multi-View Images (2019, background): “Marr-Net [27] reconstructs 3D objects by estimating depth, surface normals, and silhouettes of 2D images, which is challenging and usually leads to severe distortion [20].”
Self-Supervised Intrinsic Image Decomposition (2017, background): “Researchers have also explored reconstructing full 3D shapes through intrinsic images by making use of richer generative models [Kar et al., 2015, Wu et al., 2017].”