Learning to Infer and Execute 3D Shape Programs

By Yonglong Tian, Andrew Luo, Xingyuan Sun, Kevin Ellis, William T. Freeman, Josh Tenenbaum, Jiajun Wu. 2019

162 citations recorded by the source corpus.

Human perception of 3D shapes goes beyond reconstructing them as a set of points or a composition of geometric primitives: we also effortlessly understand higher-level shape structure such as the repetition and reflective symmetry of object parts. In contrast, recent advances in 3D shape sensing focus more on low-level geometry but less on these higher-level relationships. In this paper, we propose 3D shape programs, integrating bottom-up recognition systems with top-down, symbolic program structure to capture both low-level geometry and high-level structural priors for 3D shapes. Because there are no annotations of shape programs for real shapes, we develop neural modules that not only learn to infer 3D shape programs from raw, unannotated shapes, but also to execute these programs for shape reconstruction. After initial bootstrapping, our end-to-end differentiable model learns 3D shape programs by reconstructing shapes in a self-supervised manner. Experiments demonstrate that our model accurately infers and executes 3D shape programs for highly complex shapes from various categories. It can also be integrated with an image-to-shape module to infer 3D shape programs directly from an RGB image, leading to 3D shape reconstructions that are both more accurate and more physically plausible.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Reasoning Like Program Executors (2022, background): “(Andreas et al., 2016), reading comprehension (Gupta et al., 2019; Khot et al., 2021), knowledge base question answering (Ren et al., 2021) and 3D rendering (Tian et al., 2019).”
MONet: Unsupervised Scene Decomposition and Representation (2019, background): “Methods in this camp [Tian et al., 2019, Yao et al., 2018] make domain-speciﬁc assumptions about the latent code [Kulkarni et al., 2015] or generative process [Wu et al., 2017] to keep inference tractable, but these assumptions tend to be too strong for the methods to be broadly useful.”
Mesh R-CNN (2019, background): “Other approaches predict novel 3D shapes as sets of 3D points [8, 34], patches [15, 70], or geometric primitives [9, 64, 67]; others use deep networks to model signed distance functions [42].”
LEGO: Latent Execution-Guided Reasoning for Multi-Hop Question Answering on Knowledge Graphs (2021, background): “…some recent works propose to learn neural networks to represent a program executor, including reading comprehension (Gupta et al., 2019), visual question answering (Andreas et al., 2016), 3D rendering (Tian et al., 2019), and short programs with loops and higher-order functions (Nye et al., 2020a).”
DeepCAD: A Deep Generative Network for Computer-Aided Design Models (2021, background): “3D shapes using their proposed domain speciﬁc languages (DSLs) [39, 41, 30, 20].”
Modeling question asking using neural program generation (2019, background): “strings in Microsoft Excel sheets [4], understanding and constructing 3D scenes [13] and objects [25].”