StarCraft II: A New Challenge for Reinforcement Learning

By Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Küttler, John P. Agapiou, Julian Schrittwieser and 15 more. 2017

971 citations recorded by the source corpus.

This paper introduces SC2LE (StarCraft II Learning Environment), a reinforcement learning environment based on the StarCraft II game. This domain poses a new grand challenge for reinforcement learning, representing a more difficult class of problems than considered in most prior work. It is a multi-agent problem with multiple players interacting; there is imperfect information due to a partially observed map; it has a large action space involving the selection and control of hundreds of units; it has a large state space that must be observed solely from raw input feature planes; and it has delayed credit assignment requiring long-term strategies over thousands of steps. We describe the observation, action, and reward specification for the StarCraft II domain and provide an open source Python-based interface for communicating with the game engine. In addition to the main game maps, we provide a suite of mini-games focusing on different elements of StarCraft II gameplay. For the main game maps, we also provide an accompanying dataset of game replay data from human expert players. We give initial baseline results for neural networks trained from this data to predict game outcomes and player actions. Finally, we present initial baseline results for canonical deep reinforcement learning agents applied to the StarCraft II domain. On the mini-games, these agents learn to achieve a level of play that is comparable to a novice player. However, when trained on the main game, these agents are unable to make significant progress. Thus, SC2LE offers a new and challenging environment for exploring deep reinforcement learning algorithms and architectures.

Subfields: Artificial Intelligence, Sociology and Political Science, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Sociology and Political Science

In Reinforcement Learning

Citations in the corpus

PyTorch: An Imperative Style, High-Performance Deep Learning Library (2019, background): “In a surprisingly short amount of time, machine learning grew from recognizing individual digits [27] into autonomously playing StarCraft [28].”
Deep Reinforcement Learning that Matters (2017, background): “…experienced dramatic growth in attention and interest due to promising results in areas like: controlling continuous systems in robotics (Lillicrap et al. 2015a), playing Go (Silver et al. 2016), Atari (Mnih et al. 2013), and competitive video games (Vinyals et al. 2017; Silva and Chaimowicz 2017).”
Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2020, methodology): “It is built on the popular real-time strategy game StarCraft II and makes use of the SC2LE environment (Vinyals et al., 2017).”
Relational Deep Reinforcement Learning (2018, methodology): “We added a 2D-ConvLSTM immediately downstream of the residual blocks, to give the agent the ability to deal with recent history.”
Population Based Training of Neural Networks (2017, methodology): “…of deep reinforcement learning, showing how PBT can be used to optimise UNREAL (Jaderberg et al., 2016) on DeepMind Lab levels (Beattie et al., 2016), Feudal Networks (Vezhnevets et al., 2017) on Atari games (Bellemare et al., 2013), and simple A3C agents for StarCraft II (Vinyals et al., 2017).”
Hyperparameter Selection for Imitation Learning (2021, background): “This is the case if the goal is to design a policy that performs well in a single environment (Silver et al., 2016; Tesauro, 1995; Vinyals et al., 2017).”
Reinforcement Learning from Imperfect Demonstrations (2018, background): “, 2016) and intelligent agents in Starcraft II and vizDoom (Vinyals et al., 2017; Kempka et al., 2016).”
Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games (2017, methodology): “Google DeepMind released a game interface based on StarCraft II and claimed that it is hard to make significant progress on the full game even with the state-ofthe-art RL algorithms (Vinyals et al. 2017).”
Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomization (2021, background): “…s, a ) can be written as a 2-dimensional vector [ (cid:107) L T − L A (cid:107) 2 , I ( L T = L A )] ; in real-time strategy games (Wu & Tian, 2016; Vinyals et al., 2017; OpenAI et al., 2019), φ is typically related to the bonus points for destroying each type of units; in robotics manipulation…”
Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge? (2020, background): “SMAC consists of a diverse set of StarCraft II [30, SCII] unit micromanagement tasks of varying difﬁculty, where a collaborative team of SCII units needs to defeat an enemy team of units controlled by the built-in AI.”
Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks (2020, methodology): “This environment implements a variety of micromanagement tasks based on the popular realtime strategy game StarCraft II5 and makes use of the StarCraft II Learning Environment (SC2LE) [Vinyals et al., 2017].”
Google Research Football: A Novel Reinforcement Learning Environment (2019, background): “…progress in this ﬁeld has been fueled by making agents play games such as the iconic Atari console games (Bellemare et al. 2013; Mnih et al. 2013), the ancient game of Go (Silver et al. 2016), or professionally played video games like Dota 2 (OpenAI 2019) or Starcraft II (Vinyals et al. 2017).”
The StarCraft Multi-Agent Challenge (2019, methodology): “…al., 2016; Foerster et al., 2017, 2018a; Rashid et al., 2018; Nardelli et al., 2018; Hu et al., 2018; Shao et al., 2018; Foerster et al., 2018b) with the release of TorchCraft (Synnaeve et al., 2016) and SC2LE (Vinyals et al., 2017), interfaces to respectively StarCraft: BroodWar and StarCraft II .”
Multi-Agent Common Knowledge Reinforcement Learning (2018, methodology): “, 2017), as well as in simulated benchmarks such as StarCraft II (Vinyals et al., 2017).”
MAgent: A Many-Agent Reinforcement Learning Platform for Artificial Collective Intelligence (2017, methodology): “2017) and SC2LE (Vinyals et al. 2017) fail to meet the demand.”
Dopamine: A Research Framework for Deep Reinforcement Learning (2018, background): “…conclusions extend to similar environments, including continuous control (Duan et al., 2016; Tassa et al., 2018), ﬁrst-person environments (Kempka et al., 2016; Beattie et al., 2016; Johnson et al., 2016), and to some extent real-time strategy games (Tian et al., 2017; Vinyals et al., 2017).”
Deep Communicating Agents for Abstractive Summarization (2018, background): “Compared to prior multi-agent works on logic puzzles (Foerster et al., 2017), language learning (Lazaridou et al., 2016; Mordatch and Abbeel, 2017) and starcraft games (Vinyals et al., 2017), we present the ﬁrst study in using this framework for long text generation.”
Building Generalizable Agents with a Realistic and Rich 3D Environment (2018, background): “…of environments closely related to House3D, there are more sim-2 ulated environments such as OpenAI Gym (Brockman et al., 2016), ParlAI (Miller et al., 2017) for language communication as well as some strategic game environments (Synnaeve et al., 2016; Tian et al., 2017; Vinyals et al., 2017).”
A Study on Overfitting in Deep Reinforcement Learning (2018, background): “Significant progress is seen in many RL problems ranging from board games like Go (Silver et al., 2016, 2017b), Chess and Shogi (Silver et al., 2017a), video games like Atari (Mnih et al., 2015) and StarCraft (Vinyals et al., 2017), to real world robotics and control tasks (Lillicrap et al., 2016).”