Learning Shape Templates With Structured Implicit Functions

By Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna, William T. Freeman, Thomas A. Funkhouser. 2019

432 citations recorded by the source corpus.

Template 3D shapes are useful for many tasks in graphics and vision, including fitting observation data, analyzing shape collections, and transferring shape attributes. Because of the variety of geometry and topology of real-world shapes, previous methods generally use a library of hand-made templates. In this paper, we investigate learning a general shape template from data. To allow for widely varying geometry and topology, we choose an implicit surface representation based on composition of local shape elements. While long known to computer graphics, this representation has not yet been explored in the context of machine learning for vision. We show that structured implicit functions are suitable for learning and allow a network to smoothly and simultaneously fit multiple classes of shapes. The learned shape template supports applications such as shape exploration, correspondence, abstraction, interpolation, and semantic segmentation from an RGB image.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Generative Models, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Generative Models

In 3D Vision & Pose

Citations in the corpus

Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains (2020, background): “…volume density [27], occupancy [24], and signed distance [32], and have achieved state-of-the-art results across a variety of tasks such as shape representation [9, 10, 12, 13, 17, 26, 32], texture synthesis [15, 31], shape inference from images [22, 23], and novel view synthesis [27, 29, 35, 37].”
Learning Gradient Fields for Shape Generation (2020, neutral): “They also allow for learning a structured decomposition of shapes, representing local regions with Gaussian functions [21,22] or other primitives [58,53,28].”
SAL: Sign Agnostic Learning of Shapes From Raw Data (2019, background): “More flexibility and potentially more efficient use of the degrees of freedom of the model are achieved when the implicit function f is represented as a neural network [12, 30, 28, 2, 17].”
Local Deep Implicit Functions for 3D Shape (2019, methodology): “Meanwhile, new methods are emerging for learning to infer structured decomposition of shapes [39, 13].”
Convolutional Occupancy Networks (2020, methodology): “Implicit models have been adopted for learning implicit representations from images [23, 24, 29, 41], for encoding texture information [30], for 4D reconstruction [28] as well as for primitive-based reconstruction [10, 11, 15, 32].”
RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs (2021, background): “…coordinate-based neural representations [7, 33, 34, 44] have become a popular representation for various tasks such as 3D reconstruction [1,7,13,14,33,39,42,44,45,48,51,55,57], 3D-aware generative modelling [5, 9, 15, 16, 32, 37, 38, 41, 49, 64], and novel-view synthesis [2, 3, 12, 22, 24,…”
Deep Local Shapes: Learning Local SDF Priors for Detailed 3D Reconstruction (2020, methodology): “Structured implicit functions used anisotropic Gaussian kernels to compose global implicit shape representations [20].”
Differentiable Volumetric Rendering: Learning Implicit 3D Representations Without 3D Supervision (2019, methodology): “the representation they use as voxel-based [8, 13, 19, 61, 64, 73, 82, 83], pointbased [2, 17, 31, 40, 77, 85], mesh-based [24, 32, 41, 55, 80], or implicit representations [3, 12, 22, 30, 48, 49, 56, 66, 81].”
Local Implicit Grid Representations for 3D Scenes (2020, background): “Most deep learning studies have investigated object reconstruction, with input either as an RGB/D image [5, 37, 28, 4, 13, 10, 14] or 3D points [29, 26, 22], and yet few have considered learning to reconstruct full scenes.”
Neural Radiance Flow for 4D View Synthesis and Video Processing (2020, positive): “Recently, neural continuous implicit fields [9, 34, 58, 17, 77, 50, 48, 40, 62] have been developed to address the discretization issues and limited resolution of classical 3D representations such as voxel grids [5, 39, 11, 76, 56, 73, 72], point clouds [53, 13, 54, 14] and meshes [18, 69, 28, 26,…”
pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis (2020, background): “Emerging neural implicit scene representations promise 3D-structureaware, continuous, memory-efficient representations for parts [13, 12], objects [46, 38, 1, 16, 66, 7, 5], or scenes [10, 57, 20, 47, 56].”
Occupancy Flow: 4D Reconstruction by Learning Particle Dynamics (2019, background): “In particular, recent continuous representations [13, 22, 28, 38, 39,46,56,72] achieve impressive results at limited memory costs.”
Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion (2020, background): “Implicit Functions for rigid objects: Recently, neural networks have been used to learn a continuous implicit function representing shape [43, 48, 10, 44, 35].”
GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields (2020, background): “Implicit Functions: Using implicit functions to represent 3D geometry has gained popularity in learning-based 3D reconstruction [11, 12, 22, 59, 60, 65, 67, 69, 76] and has been extended to scene-level reconstruction [8, 13, 35, 72, 79].”