Large Batch Simulation for Deep Reinforcement Learning

By Brennan Shacklett, Erik Wijmans, Aleksei Petrenko, Manolis Savva, Dhruv Batra, Vladlen Koltun, Kayvon Fatahalian. 2021

29 citations recorded by the source corpus.

We accelerate deep reinforcement learning-based training in visually complex 3D environments by two orders of magnitude over prior work, realizing end-to-end training speeds of over 19,000 frames of experience per second on a single GPU and up to 72,000 frames per second on a single eight-GPU machine. The key idea of our approach is to design a 3D renderer and embodied navigation simulator around the principle of"batch simulation": accepting and executing large batches of requests simultaneously. Beyond exposing large amounts of work at once, batch simulation allows implementations to amortize in-memory storage of scene assets, rendering work, data loading, and synchronization costs across many simulation requests, dramatically improving the number of simulated agents per GPU and overall simulation throughput. To balance DNN inference and training costs with faster simulation, we also build a computationally efficient policy DNN that maintains high task performance, and modify training algorithms to maintain sample efficiency when training with large mini-batches. By combining batch simulation and DNN performance optimizations, we demonstrate that PointGoal navigation agents can be trained in complex 3D environments on a single GPU in 1.5 days to 97% of the accuracy of agents trained on a prior state-of-the-art system using a 64-GPU cluster over three days. We provide open-source reference implementations of our batch 3D renderer and simulator to facilitate incorporation of these ideas into RL systems.

Subfields: Artificial Intelligence, Computer Science Applications, Computer Vision and Pattern Recognition, Reinforcement Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dhruv Batra
2. Jitendra Malik
3. Trevor Darrell
4. Abhinav Kumar Gupta
5. Vladlen Koltun
6. Sergey Levine
7. Kristen Grauman
8. Ruslan Salakhutdinov

In Computer Science Applications

1. Dhruv Batra
2. Vladlen Koltun
3. Sergey I. Nikolenko

In Computer Vision and Pattern Recognition

1. Jitendra Malik
2. Dhruv Batra
3. Luc Van Gool
4. Abhinav Kumar Gupta
5. Trevor Darrell
6. Kristen Grauman
7. Dacheng Tao
8. Andrew Zisserman

In Reinforcement Learning

1. Dhruv Batra
2. Jitendra Malik
3. Devendra Singh Chaplot
4. Manolis Savva
5. Vladlen Koltun
6. Sergey Levine
7. Marco Hutter
8. F. Meier

In Optimization & Training

1. Marco Hutter
2. Jitendra Malik
3. Vladlen Koltun
4. Zsolt Kira
5. Angel X. Chang
6. F. Meier
7. Andrew Szot
8. Dhruv Batra

Citations in the corpus

• Habitat 2.0: Training Home Assistants to Rearrange their Habitat (2021, background): “[80] Brennan Shacklett, Erik Wijmans, Aleksei Petrenko, Manolis Savva, Dhruv Batra, Vladlen Koltun, and Kayvon Fatahalian.”
• Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning (2021, methodology): “In the context of visual navigation, large batch simulation has been used to increase the training throughput [11].”