Dota 2 with Large Scale Deep Reinforcement Learning

By Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Christopher Hesse and 15 more. 2019

2,179 citations recorded by the source corpus.

On April 13th, 2019, OpenAI Five became the first AI system to defeat the world champions at an esports game. The game of Dota 2 presents novel challenges for AI systems such as long time horizons, imperfect information, and complex, continuous state-action spaces, all challenges which will become increasingly central to more capable AI systems. OpenAI Five leveraged existing reinforcement learning techniques, scaled to learn from batches of approximately 2 million frames every 2 seconds. We developed a distributed training system and tools for continual training which allowed us to train OpenAI Five for 10 months. By defeating the Dota 2 world champion (Team OG), OpenAI Five demonstrates that self-play reinforcement learning can achieve superhuman performance on a difficult task.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Object-Centric Learning with Slot Attention (2020, background): “[4] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al.”
Jukebox: A Generative Model for Music (2020, methodology): “To reduce computation required to train the lyrics conditional model, we use a pretrained unconditional top-level prior as our decoder and introduce the lyrics encoder using model surgery (Berner et al., 2019).”
Generative Language Modeling for Automated Theorem Proving (2020, background): “Artiﬁcial neural networks have enjoyed a spectacularly successful decade, having made considerable advances in computer vision [1, 2], translation [3, 4, 5], speech recognition [6, 7], image generation [8, 9, 10, 11, 12], game playing [13, 14, 15], and robotics [16, 17].”
Toward Causal Representation Learning (2021, background): “Machine learning models that incorporate or learn structural knowledge of an environment have been shown to be more efﬁcient and generalize better [9], [11], [15], [16], [27], [58], [77], [84], [85], [141], [157], [177], [181], [197], [211], [212], [244], [258], [272], [274].”
The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games (2021, background): “Recent advances in reinforcement learning (RL) and multi-agent reinforcement learning (MARL) have led to a great deal of progress in creating artificial agents which can cooperate to solve tasks: DeepMind’s AlphaStar surpassed professional-level performance in the StarCraft II [34], OpenAI Five defe”
Towards Causal Representation Learning (2021, background): “accumulating evidence supporting the usefulness of an appropriate structured representation of the environment [2, 26, 258].”
CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning (2020, background): “…in domains such as game playing (Mnih et al., 2013; Silver et al., 2017), learning motor control for high-dimensional simulated robots (Akkaya et al., 2019), multi-agent settings (Baker et al., 2019; Berner et al., 2019) and for studying transfer in the context of meta-learning (Yu et al., 2019).”
Regularized Behavior Value Estimation (2021, methodology): “This paradigm has attained considerable success on Atari (Mnih et al., 2015), Go (Silver et al., 2017), StarCraft II and Dota 2 (Vinyals et al., 2019; Berner et al., 2019), and robotics (Andrychowicz et al., 2020).”
Hyperparameter Selection for Imitation Learning (2021, background): “Introduction Recent advances in Reinforcement Learning (RL) now allow optimizing control policies with respect to a given reward function even for high-dimensional observation and action spaces (Berner et al., 2019; Vinyals et al., 2019).”
RL Unplugged: Benchmarks for Offline Reinforcement Learning (2020, background): “Reinforcement Learning (RL) has seen important breakthroughs, including learning directly from raw sensory streams (Mnih et al., 2015), solving long-horizon reasoning problems such as Go (Silver et al., 2016), StarCraft II (Vinyals et al., 2019), DOTA (Berner et al., 2019), and learning motor contro”
Acme: A Research Framework for Distributed Reinforcement Learning (2020, background): “In pursuit of this objective, the scale and complexity of agents developed by the research community has grown over time; OpenAI Five (Berner et al., 2019) and AlphaStar (Vinyals et al., 2019) serve as just two recent examples of this phenomenon.”
TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrations (2021, neutral): “Deep reinforcement learning (DRL) has shown great success in many video games, including the Atari games (Mnih et al., 2013), StarCraft II (Vinyals et al., 2019), Dota II (Berner et al., 2019), etc.”
Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomization (2021, background): “…as a 2-dimensional vector [ (cid:107) L T − L A (cid:107) 2 , I ( L T = L A )] ; in real-time strategy games (Wu & Tian, 2016; Vinyals et al., 2017; OpenAI et al., 2019), φ is typically related to the bonus points for destroying each type of units; in robotics manipulation (Levine et al.,…”
RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, background): “Multi-Agent Reinforcement Learning (MARL) Many challenging tasks (Vinyals et al., 2019; Berner et al., 2019; Samvelyan et al., 2019; Jaderberg et al., 2019) can be modelled as multi-agent learning problems.”
Coordination Among Neural Modules Through a Shared Global Workspace (2021, background): “The induced structure, and separation of knowledge has improved generalization, model-size scaling, and longrange dependencies (Berner et al., 2019; Vinyals et al., 2019; Brown et al., 2020).”