GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learning

By Jacky Liang, Viktor Makoviychuk, Ankur Handa, Nuttapong Chentanez, Miles Macklin, Dieter Fox. 2018

245 citations recorded by the source corpus.

Most Deep Reinforcement Learning (Deep RL) algorithms require a prohibitively large number of training samples for learning complex tasks. Many recent works on speeding up Deep RL have focused on distributed training and simulation. While distributed training is often done on the GPU, simulation is not. In this work, we propose using GPU-accelerated RL simulations as an alternative to CPU ones. Using NVIDIA Flex, a GPU-based physics engine, we show promising speed-ups of learning various continuous-control, locomotion tasks. With one GPU and CPU core, we are able to train the Humanoid running task in less than 20 minutes, using 10-1000x fewer CPU cores than previous works. We also demonstrate the scalability of our simulator to multi-GPU settings to train more challenging locomotion tasks.

Subfields: Artificial Intelligence, Biomedical Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

Citations in the corpus

Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience (2018, methodology): “The RL training is performed on a GPU based simulator using a parallelized version of proximal policy optimization (PPO) [36] on a multi-GPU cluster [37].”
Alternative Paths Planner (APP) for Provably Fixed-time Manipulation Planning in Semi-structured Environments (2020, background): “1), both in simulation (Isaac Gym [19]) and in the real world.”
Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning (2021, background): “This approach was taken by [13], and showed very promising results running simulation on GPU, proving that it is possible to greatly reduce both training time as well as computational resources required to solve very challenging tasks using RL.”
Learning to Compose Hierarchical Object-Centric Controllers for Robotic Manipulation (2020, methodology): “[39] J.”
DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames (2019, methodology): “Several works have proposed systems for distributed RL (Heess et al., 2017; Liang et al., 2018a; Tian et al., 2019; Silver et al., 2016; OpenAI, 2018; Espeholt et al., 2018).”
Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning (2021, methodology): “We build upon [10, 12] by pushing the parallelization further, optimizing the training algorithm, and applying the approach to a challenging real-world robotics task.”