Matryoshka Networks: Predicting 3D Geometry via Nested Shape Layers

By Stephan R. Richter, Stefan Roth. 2018

144 citations recorded by the source corpus.

In this paper, we develop novel, efficient 2D encodings for 3D geometry, which enable reconstructing full 3D shapes from a single image at high resolution. The key idea is to pose 3D shape reconstruction as a 2D prediction problem. To that end, we first develop a simple baseline network that predicts entire voxel tubes at each pixel of a reference view. By leveraging well-proven architectures for 2D pixel-prediction tasks, we attain state-of-the-art results, clearly outperforming purely voxel-based approaches. We scale this baseline to higher resolutions by proposing a memory-efficient shape encoding, which recursively decomposes a 3D shape into nested shape layers, similar to the pieces of a Matryoshka doll. This allows reconstructing highly detailed shapes with complex topology, as demonstrated in extensive experiments; we clearly outperform previous octree-based approaches despite having a much simpler architecture using standard network components. Our Matryoshka networks further enable reconstructing shapes from IDs or shape similarity, as well as shape sampling.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Deep Level Sets: Implicit Surface Representations for 3D Shape Inference (2019, background): “A majority of existing 3D representation learning approaches are based on voxel occupancy [3,12,28,29] but a considerable amount of attention has also been put on point clouds [10,27] and explicit shape parameterisation [21].”
Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era (2019, methodology): “To achieve high resolution 3D volumetric reconstruction, many recent papers use intermediation, through multiple depth maps, followed by volumetric [38], [46], [51], [92] or point-based [80] fusion.”
Deep Stereo Using Adaptive Thin Volume Representation With Uncertainty Awareness (2019, background): “Volumetric based approaches partition the space into a regular 3D volume with millions of small voxels [23, 27, 54, 55, 60, 40], and the network predicts if a voxel is on the surface or not.”
Implicit Surface Representations As Layers in Neural Networks (2019, methodology): “For comparison we also include results from two additional existing methods, R2N2 [3] and Matryoschka networks [30], code provided by the authors.”
What Do Single-View 3D Reconstruction Networks Learn? (2019, methodology): “47 IoU) [37]) do not actually perform reconstruction but image classification.”
Pixel2Mesh++: Multi-View 3D Mesh Generation via Deformation (2019, background): “Some other representations have been proposed, such as geometry images [33], depth images [36, 31], classification boundaries [26, 3], signed distance function [28], etc.”
Mesh R-CNN (2019, methodology): “Some methods predict regular voxel grids [5, 71, 72]; while intuitive, scaling to high-resolution outputs requires complex octree [50, 62] or nested shape architectures [49].”
DUSt3R: Geometric 3D Vision Made Easy (2023, unknown): “Shapes can be stored as occupancy in regular grids [19, 98, 112, 118, 138, 154, 155], octree structures [128], collections of parametric surface elements [39], point clouds encoders [29, 66, 67], free-form deformation of template meshes [89] or per-view depthmaps [54].”