Sample Factory: Egocentric 3D Control from Pixels at 100000 FPS with Asynchronous Reinforcement Learning

By Aleksei Petrenko, Zhehui Huang, T. K. Satish Kumar, G. Sukhatme, Vladlen Koltun. 2020

137 citations recorded by the source corpus.

Increasing the scale of reinforcement learning experiments has allowed researchers to achieve unprecedented results in both training sophisticated agents for video games, and in sim-to-real transfer for robotics. Typically such experiments rely on large distributed systems and require expensive hardware setups, limiting wider access to this exciting area of research. In this work we aim to solve this problem by optimizing the efficiency and resource utilization of reinforcement learning algorithms instead of relying on distributed computation. We present the "Sample Factory", a high-throughput training system optimized for a single-machine setting. Our architecture combines a highly efficient, asynchronous, GPU-based sampler with off-policy correction techniques, allowing us to achieve throughput higher than $10^5$ environment frames/second on non-trivial control problems in 3D without sacrificing sample efficiency. We extend Sample Factory to support self-play and population-based training and apply these techniques to train highly capable agents for a multiplayer first-person shooter game. The source code is available at this https URL

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dhruv Batra
2. Abhinav Kumar Gupta
3. Trevor Darrell
4. Jitendra Malik
5. Kristen Grauman
6. Vladlen Koltun
7. Sergey Levine
8. Ruslan Salakhutdinov

In Computer Vision and Pattern Recognition

1. Dhruv Batra
2. Jitendra Malik
3. Abhinav Kumar Gupta
4. Luc Van Gool
5. Kristen Grauman
6. Trevor Darrell
7. Dacheng Tao
8. Andrew Zisserman

In Reinforcement Learning

1. Dhruv Batra
2. Erik Wijmans
3. Devendra Singh Chaplot
4. Manolis Savva
5. Jitendra Malik
6. Vladlen Koltun
7. Sergey Levine
8. F. Meier

In ML Systems & Hardware

1. Iuri Frosio
2. Vladlen Koltun
3. Dhruv Batra
4. Brennan Shacklett

Citations in the corpus

• Habitat 2.0: Training Home Assistants to Rearrange their Habitat (2021, background): “[79] Aleksei Petrenko, Zhehui Huang, Tushar Kumar, Gaurav Sukhatme, and Vladlen Koltun.”
• Gymnasium: A Standard Interface for Reinforcement Learning Environments (2025, background): “We expand on this in the following section.”
• Large Batch Simulation for Deep Reinforcement Learning (2021, background): “, by “centralizing” computations to facilitate efficient batch execution on GPUs or TPUs (Espeholt et al., 2020; Petrenko et al., 2020) or by parallelizing across GPUs (Wijmans et al.”
• How to Train PointGoal Navigation Agents on a (Sample and Compute) Budget (2020, unknown): “Compute efficiency has been studied in model-free RL in the context of distributed systems methods [16, 40, 41, 42].”
• Accelerating Reinforcement Learning through GPU Atari Emulation (2019, methodology): “In contrast with CuLE, Sample Factory uses CPU simulation and a novel GPU-based sampler; like in the case of CuLE and other high throughput systems [6, 19], off-policy correction is then needed to process the large amount of generated data.”