Learning To Simulate

By Nataniel Ruiz, Samuel Schulter, Manmohan Chandraker. 2018

123 citations recorded by the source corpus.

Simulation is a useful tool in situations where training data for machine learning models is costly to annotate or even hard to acquire. In this work, we propose a reinforcement learning-based method for automatically adjusting the parameters of any (non-differentiable) simulator, thereby controlling the distribution of synthesized data in order to maximize the accuracy of a model trained on that data. In contrast to prior art that hand-crafts these simulation parameters or adjusts only parts of the available parameters, our approach fully controls the simulator with the actual underlying goal of maximizing accuracy, rather than mimicking the real data distribution or randomly generating a large volume of data. We find that our approach (i) quickly converges to the optimal simulation parameters in controlled experiments and (ii) can indeed discover good sets of parameters for an image rendering simulator in actual computer vision applications.

Subfields: Developmental and Educational Psychology, Safety Research, Reinforcement Learning, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Developmental and Educational Psychology

In Safety Research

In Reinforcement Learning

In Generative Models

Citations in the corpus

AutoML: A Survey of the State-of-the-Art (2019, methodology): “Furthermore, a reinforcement learning-based method is proposed in [46] for optimizing the parameters of data simulator to control the distribution of the synthesized data.”
Solving Rubik's Cube with a Robot Hand (2019, neutral): “For policy training, one approach viewed the problem as a bi-level optimization [115, 89].”
Meta-Learning in Neural Networks: A Survey (2020, methodology): “10, one can also train the graphics engine [159] or simulator [160] so as to optimize the real-data (validation) performance of the downstream model after training on data generated by that environment simulator.”
Informed Machine Learning – A Taxonomy and Survey of Integrating Prior Knowledge into Learning Systems (2019, unknown): “[80], [81] to be informed machine learning.”
Increasing generality in machine learning through procedural content generation (2019, background): “We identify ﬁve main open challenges that we believe are essential in pushing the ﬁeld of PCG forward and to realizing its promise to create more adaptive and lifelong learning ML agents.”
Active Domain Randomization (2019, methodology): “Chebotar et al. (2018) and Ruiz et al. (2018) use RL for effective transfer by learning parameters of a simulation that accurately describes the target domain, but require the target domain for reward calculation, which can lead to overﬁtting.”
Synthetic Data for Deep Learning (2019, background): “[507], who also learn a policy πω that outputs the parameters for a simulator, learning to generate data to maximize validation accuracy, with reinforcement learning techniques, and Louppe et al.”
StyleGAN2 Distillation for Feed-forward Image Manipulation (2020, background): “To perform well on real world images, this data need to overcome both appearance gap [22, 15, 51, 52, 49] and content gap [28, 46].”
How to pick the domain randomization parameters for sim-to-real transfer of reinforcement learning policies? (2019, result): “Furthermore, our result is consistent with ongoing research in domain randomization for supervised learning which demonstrated the importance of the sampling distribution for sim-to-real transfer success ([28], [29]).”
Meta-Sim: Learning to Generate Synthetic Datasets (2019, methodology): “[39] optimize high level exposed parameters by optimizing for downstream task performance using Reinforcement Learning.”