Meta-Sim: Learning to Generate Synthetic Datasets

By Amlan Kar, Aayush Prakash, Ming-Yu Liu, Eric Cameracci, Justin Yuan, Matt Rusiniak, David Acuna, Antonio Torralba, Sanja Fidler. 2019

279 citations recorded by the source corpus.

Training models to high-end performance requires availability of large labeled datasets, which are expensive to get. The goal of our work is to automatically synthesize labeled datasets that are relevant for a downstream task. We propose Meta-Sim, which learns a generative model of synthetic scenes, and obtain images as well as its corresponding ground-truth via a graphics engine. We parametrize our dataset generator with a neural network, which learns to modify attributes of scene graphs obtained from probabilistic scene grammars, so as to minimize the distribution gap between its rendered outputs and target data. If the real dataset comes with a small labeled validation set, we additionally aim to optimize a meta-objective, i.e. downstream task performance. Experiments show that the proposed method can greatly improve content generation quality over a human-engineered probabilistic scene grammar, both qualitatively and quantitatively as measured by performance on a downstream task.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Meta & Continual Learning

Citations in the corpus

Occluded Video Instance Segmentation: A Benchmark (2021, background): “2014) with partial occlusions, and some works (Nikolenko, 2019; Kar et al., 2019; Devaranjan et al., 2020) synthesize structured amodal data in occluded scenes using simulators.”
Solving Rubik's Cube with a Robot Hand (2019, background): “For vision models, domain randomization has been modified to improve image content diversity [50, 85, 21] and to adapt the distribution by using an adversarial network [120].”
Meta-Learning in Neural Networks: A Survey (2020, methodology): “This paradigm has been used in RL [160] as well as vision [159], [273].”
DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort (2021, unknown): “Generative Models of Labeled Data: Prior work on dataset synthesis has mainly focused on generative models of 3D scene graphs, utilizing graphics to render images and their labels [24, 12, 30].”
Increasing generality in machine learning through procedural content generation (2019, background): “In addition to developing more sophisticated machine learning models, one important research challenge in crossing the reality gap is the content gap [90].”
Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization (2021, methodology): “Conceptually, [84] trains a generator together with a pixel-wise discriminator network to perform segmentation, while [41, 17, 52] learn to generate synthetic 3D scenes by matching distributions of real and rendered imagery.”
Synthetic Data for Deep Learning (2019, methodology): “First, the Meta-Sim framework [309], which we discussed in Section 5.”
Meta-Sim2: Unsupervised Learning of Scene Structure for Synthetic Data Generation (2020, methodology): “We define scene grammars and represent strings sampled from the grammar as scene graphs following [48,30] (see Fig.”
Block-NeRF: Scalable Large Scene Neural View Synthesis (2022, unknown): “[26] and Devaranjan et al .”
StyleGAN2 Distillation for Feed-forward Image Manipulation (2020, background): “To perform well on real world images, this data need to overcome both appearance gap [22,15,51,52,49] and content gap [28,46].”
Fake it till you make it: face analysis in the wild using synthetic data alone (2021, methodology): “The reason for this is the domain gap – a difference in distributions between real and synthetic data which makes generalization difficult [25].”
AutoFlow: Learning a Better Training Set for Optical Flow (2021, background): “hard to scale, and quite a few synthetic datasets have been developed [39, 9, 11, 22].”
BigDatasetGAN: Synthesizing ImageNet with Pixel-wise Annotations (2022, background): “Recent works also propose to learn the parameters of the data generation pipeline to reduce the distribution gap [14,36,46].”