Active Domain Randomization

By Bhairav Mehta, Manfred Diaz, Florian Golemo, C. Pal, Liam Paull. 2019

338 citations recorded by the source corpus.

Domain randomization is a popular technique for improving domain transfer, often used in a zero-shot setting when the target domain is unknown or cannot easily be used for training. In this work, we empirically examine the effects of domain randomization on agent generalization. Our experiments show that domain randomization may lead to suboptimal, high-variance policies, which we attribute to the uniform sampling of environment parameters. We propose Active Domain Randomization, a novel algorithm that learns a parameter sampling strategy. Our method looks for the most informative environment variations within the given randomization ranges by leveraging the discrepancies of policy rollouts in randomized and reference environment instances. We find that training more frequently on these instances leads to better overall agent generalization. In addition, when domain randomization and policy transfer fail, Active Domain Randomization offers more insight into the deficiencies of both the chosen parameter ranges and the learned policy, allowing for more focused debugging. Our experiments across various physics-based simulated and a real-robot task show that this enhancement leads to more robust, consistent policies.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Solving Rubik's Cube with a Robot Hand (2019, methodology): “In [68], a discriminator was used to guide the distribution in simulation.”
BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning (2022, background): “This includes works that generalize skills to novel objects [31, 32, 33, 34, 35], to novel environments [36], from simulation to reality [37, 38, 39, 40, 41], and to new manipulation skills and objects [42, 43, 44, 45].”
Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning (2021, background): “In (108), a discriminator is adversarially trained and used to guide the randomization process that generates systems that are less explored or exploited by the current policy.”
Multi-Agent Manipulation via Locomotion using Hierarchical Sim2Real (2019, background): “Another work [10] uses meta-learning to learn a small set of randomization settings which simultaneously maximize the utility of the learned policy and the diversity of the settings.”
Synthetic Data for Deep Learning (2019, background): “VADRA works much better for the syn-to-real transfer on problems such as object detection and segmentation than regular domain randomization. Similar ideas have been recently explored by Mehta et al. [406], who present Active Domain Randomization, again learning a policy for generating better simula”
Robust Domain Randomised Reinforcement Learning through Peer-to-Peer Distillation (2020, neutral): “However, in practice, diverse training domains lead to high variance in gradients [12], thus leading to poor learning behaviour.”
Variational Automatic Curriculum Learning for Sparse-Reward Cooperative Multi-Agent Problems (2021, background): “[10, 11] assume a ﬁxed goal and generate tasks with starting states increasingly far away from the goal; [1, 21] consider a task space over simulator conﬁgurations and train from the easiest settings to the hardest for sim-to-real adaptation.”
Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design (2020, background): “[23] propose to enhance DR by learning which parameter values lead to the biggest decrease in performance compared to a reference environment.”
Robust Visual Domain Randomization for Reinforcement Learning. (2019, background): “However, domain randomization has been empirically shown to often lead to suboptimal policies with high variance in performance over different randomizations (Mehta et al., 2019).”
Automatic Curriculum Learning For Deep RL: A Short Survey (2020, positive): “ACL can also help agents to generalize from simulation settings to the real world (Sim2Real) [OpenAI et al. , 2019; Mehta et al. , 2019] or to maximize performance and robustness in multi-agent settings via Self-Play [Silver et al. , 2017; Pinto et al. , 2017; Bansal et al. , 2017; Baker et al. ,…”