Meta-Sim2: Unsupervised Learning of Scene Structure for Synthetic Data Generation

By Jeevan Devaranjan, Amlan Kar, Sanja Fidler. 2020

97 citations recorded by the source corpus.

Procedural models are being widely used to synthesize scenes for graphics, gaming, and to create (labeled) synthetic datasets for ML. In order to produce realistic and diverse scenes, a number of parameters governing the procedural models have to be carefully tuned by experts. These parameters control both the structure of scenes being generated (e.g. how many cars in the scene), as well as parameters which place objects in valid configurations. Meta-Sim aimed at automatically tuning parameters given a target collection of real images in an unsupervised way. In Meta-Sim2, we aim to learn the scene structure in addition to parameters, which is a challenging problem due to its discrete nature. Meta-Sim2 proceeds by learning to sequentially sample rule expansions from a given probabilistic scene grammar. Due to the discrete nature of the problem, we use Reinforcement Learning to train our model, and design a feature space divergence between our synthesized and target images that is key to successful training. Experiments on a real driving dataset show that, without any supervision, we can successfully learn to generate data that captures discrete structural statistics of objects, such as their frequency, in real images. We also show that this leads to downstream improvement in the performance of an object detector trained on our generated dataset as opposed to other baseline simulation methods. Project page: this https URL.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Representation & Self-Supervised Learning

Citations in the corpus

Occluded Video Instance Segmentation: A Benchmark (2021, background): “2014) with partial occlusions, and some works (Nikolenko, 2019; Kar et al., 2019; Devaranjan et al., 2020) synthesize structured amodal data in occluded scenes using simulators.”
A Scaling Law for Syn2real Transfer: How Much Is Your Pre-training Effective? (2021, unknown): “The utility of synthetic images as supervised data for computer vision tasks has been continuously studied by many researchers [9, 12, 14, 22, 29, 31, 43–45, 56, 61].”
DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort (2021, unknown): “Generative Models of Labeled Data: Prior work on dataset synthesis has mainly focused on generative models of 3D scene graphs, utilizing graphics to render images and their labels [24, 12, 30].”
Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization (2021, methodology): “Conceptually, [84] trains a generator together with a pixel-wise discriminator network to perform segmentation, while [41, 17, 52] learn to generate synthetic 3D scenes by matching distributions of real and rendered imagery.”
BigDatasetGAN: Synthesizing ImageNet with Pixel-wise Annotations (2022, background): “Recent works also propose to learn the parameters of the data generation pipeline to reduce the distribution gap [14,36,46].”
Block-NeRF: Scalable Large Scene Neural View Synthesis (2022, background): “[12] proposed to minimize the scene-level distribution shift from rendered outputs to real camera sensor data through a learned scenario generation framework.”