Configurable 3D Scene Synthesis and 2D Image Rendering with Per-pixel Ground Truth Using Stochastic Grammars

By Chenfanfu Jiang, Siyuan Qi, Yixin Zhu, Siyuan Huang, Jenny Lin, Lap-Fai Craig Yu, Demetri Terzopoulos, Song-Chun Zhu. 2017

86 citations recorded by the source corpus.

We propose a systematic learning-based approach to the generation of massive quantities of synthetic 3D scenes and arbitrary numbers of photorealistic 2D images thereof, with associated ground truth information, for the purposes of training, benchmarking, and diagnosing learning-based computer vision and robotics algorithms. In particular, we devise a learning-based pipeline of algorithms capable of automatically generating and rendering a potentially infinite variety of indoor scenes by using a stochastic grammar, represented as an attributed Spatial And-Or Graph, in conjunction with state-of-the-art physics-based rendering. Our pipeline is capable of synthesizing scene layouts with high diversity, and it is configurable inasmuch as it enables the precise customization and control of important attributes of the generated scenes. It renders photorealistic RGB images of the generated scenes while automatically synthesizing detailed, per-pixel ground truth data, including visible surface depth and normal, object identity, and material information (detailed to object parts), as well as environments (e.g., illuminations and camera viewpoints). We demonstrate the value of our synthesized dataset, by improving performance in certain machine-learning-based scene understanding tasks—depth and surface normal prediction, semantic segmentation, reconstruction, etc.—and by providing benchmarks for and diagnostics of trained models by modifying object attributes and scene properties in a controllable manner.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding (2020, background): “In response to this challenge, the computer vision community has developed several photorealistic synthetic datasets and interactive simulation environments that have spurred rapid progress towards the goal of holistic indoor scene understanding [5, 6, 8, 9, 13, 14, 17, 20, 22, 29, 31, 34, 35, 37, 4”
Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense (2020, background): “To reach this goal, we have seen the increasing use of synthetic data and simulation platforms for indoor scenes in recent years by leveraging state-of-the-art game engines and free, publicly available 3D content [376, 377, 288, 378], including MINOR [379], HoME [380], Gibson [381],”
Kimera: From SLAM to spatial perception with 3D dynamic scene graphs (2021, background): “2017), highlevel understanding (Choi et al. 2013; Zhao and Zhu 2013a; Huang et al. 2018b; Jiang et al. 2018), visual questionanswering (Fukui et al.”
Scenic: a language for scenario specification and scene generation (2018, background): “er approach is to give rules or a grammar specifying how the data can be generated, as in generative fuzz testing [47], procedural generation from shape grammars [34], and grammar-based scenesynthesis[24].Whilegrammarsprovidemuchgreatercontrol, they are hard to read and write and do not easily allow”
Holistic++ Scene Understanding: Single-View 3D Holistic Scene Parsing and Human Pose Estimation With Human-Object Interaction and Physical Commonsense (2019, background): “In robotics and computer vision, prior work has exploited human-object relations in event, object, and scene modeling, but most work focuses on human-object relation detection in images [5, 30, 25, 21], probabilistic modeling from multiple data sources [45, 33, 13], and snapshots generation or scene”
Learning 3D Semantic Scene Graphs From 3D Indoor Reconstructions (2020, background): “A line of works use probabilistic grammar to parse scenes [28, 53] or control scene synthesis [15].”
3D Scene Graph: A Structure for Unified Semantics, 3D Space, and Camera (2019, background): “Grammars that combine geometry, affordance and appearance have been used toward holistic scene parsing in images, where information about the scene and objects is captured in a hierarchical tree structure [11, 48, 21, 19].”