Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs

By Maxim Tatarchenko, Alexey Dosovitskiy, Thomas Brox. 2017

784 citations recorded by the source corpus.

We present a deep convolutional decoder architecture that can generate volumetric 3D outputs in a compute- and memory-efficient manner by using an octree representation. The network learns to predict both the structure of the octree, and the occupancy values of individual cells. This makes it a particularly valuable technique for generating 3D shapes. In contrast to standard decoders acting on regular voxel grids, the architecture does not have cubic complexity. This allows representing much higher resolution outputs with a limited memory budget. We demonstrate this in several application domains, including 3D convolutional autoencoders, generation of objects and whole scenes from high-level representations, and shape from a single image.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

In 3D Vision & Pose

Citations in the corpus

Occupancy Networks: Learning 3D Reconstruction in Function Space (2018, methodology): “Due to the high memory requirements of voxel representations, recent works have proposed to reconstruct 3D objects in a multi-resolution fashion [28, 67].”
Dynamic Graph CNN for Learning on Point Clouds (2018, methodology): “ptation to a grid structure. As a simple way to overcome this issue, view-based [Su et al. 2015; Wei et al. 2016] and volumetric representations [Klokov and Lempitsky 2017; Maturana and Scherer 2015; Tatarchenko et al. 2017; Wu et al. 2015]—or their combination [Qi et al. 2016]—“place” geometric dat”
DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation (2019, methodology): “Octree-based methods [49, 41, 22] alleviate the compute and memory limitations of dense voxel methods, extending for example the ability to learn at up to 512 resolution [49], but even this resolution is far from producing shapes that are visually compelling.”
Learning to Reconstruct Shapes from Unseen Classes (2018, background): “Apart from voxels, some researchers have also studied reconstructing objects in point clouds [Fan et al., 2017] or octave trees [Riegler et al., 2017, Tatarchenko et al., 2017].”
Learning Implicit Fields for Generative Shape Modeling (2018, background): “There have been a variety of 3D shape representations for deep learning of shapes, such as voxel grids [9, 14, 28, 45, 46], octrees [18, 35, 40, 43, 44], multi-view images [29, 39], point clouds [1, 12, 13, 32, 33, 47, 48], geometry images [37, 38], deformable mesh/patches [16, 38, 42, 47], and part”
Learning Shape Priors for Single-View 3D Completion and Reconstruction (2018, methodology): “The only exception is the CD on PASCAL 3D+ cars, where OGN performs the best.”
RayNet: Learning Volumetric 3D Reconstruction with Ray Potentials (2018, methodology): “, octrees) have been proposed to increase the output resolution beyond 323 voxels [12, 27, 31].”
Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images (2018, methodology): “Most recently, Tatarchenko et al. [30] have proposed an octree representation, which allows to reconstructing higher resolution outputs with a limited memory budget.”
Learning Category-Specific Mesh Reconstruction from Image Collections (2018, methodology): “Relying on ground-truth 3D supervision (using synthetic data), some approaches have examined learning voxel [4,8,39,33], point cloud [7] or octree [10,26] prediction.”
A Papier-Mache Approach to Learning 3D Surface Generation (2018, neutral): “Representing a shape as a volumetric function [6, 12, 30] only provides voxel-scale sampling of the underlying smooth and continuous surface.”
Multi-View Silhouette and Depth Decomposition for High Resolution 3D Object Representation (2018, methodology): “Octree generating networks (OGN) [44] use a convolutional network that operates directly on octrees, rather than in voxel space.”
PointRend: Image Segmentation As Rendering (2019, background): “Instead, recent works consider more efficient non-uniform representations such as meshes [42, 13], signed distance functions [32], and octrees [41].”
Neural 3D Mesh Renderer (2017, methodology): “are 3D extensions of pixels, are the most widely used format in machine learning because they can be processed by CNNs [2, 17, 20, 24, 30, 31, 34, 35, 36].”
D-NeRF: Neural Radiance Fields for Dynamic Scenes (2020, background): “Standard representations for rigid geometry include point-clouds [42, 33], voxels [13, 48] and octrees [45, 39].”
Pix2Vox: Context-Aware 3D Reconstruction From Single and Multi-View Images (2019, neutral): “Category 3D-R2N2 [2] OGN [23] DRC [25] PSGN [5] Pix2Vox-F Pix2Vox-A”
Visual Object Networks: Image Generation with Disentangled 3D Representations (2018, background): “Popular representations include voxels [Wu et al., 2016], point clouds [Gadelha et al., 2017b, Achlioptas et al., 2018], and octave trees [Tatarchenko et al., 2017].”
What Do Single-View 3D Reconstruction Networks Learn? (2019, methodology): “11 we see that OGN and Matryoshka Networks outperform Oracle NN in terms of precision.”