Learning macromanagement in starcraft from replays using deep learning

By Niels Justesen, Sebastian Risi. 2017

73 citations recorded by the source corpus.

The real-time strategy game StarCraft has proven to be a challenging environment for artificial intelligence techniques, and as a result, current state-of-the-art solutions consist of numerous hand-crafted modules. In this paper, we show how macromanagement decisions in StarCraft can be learned directly from game replays using deep learning. Neural networks are trained on 789,571 state-action pairs extracted from 2,005 replays of highly skilled players, achieving top-1 and top-3 error rates of 54.6% and 22.9% in predicting the next build action. By integrating the trained network into UAlbertaBot, an open source StarCraft bot, the system can significantly outperform the game's built-in Terran bot, and play competitively against UAlbertaBot with a fixed rush strategy. To our knowledge, this is the first time macromanagement tasks are learned directly from replays in StarCraft. While the best hand-crafted strategies are still the state-of-the-art, the deep network approach is able to express a wide range of different strategies and thus improving the network's performance further with deep reinforcement learning is an immediately promising avenue for future research. Ultimately this approach could lead to strong StarCraft bots that are less reliant on hard-coded strategies.

Subfields: Artificial Intelligence, Sociology and Political Science

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Sociology and Political Science

Citations in the corpus

StarCraft II: A New Challenge for Reinforcement Learning (2017, background): “, [22, 36]) or learning game outcome or build orders from replay data [9, 12].”
On Reinforcement Learning for Full-length Game of StarCraft (2018, background): “T, the exploration is also over. Typical RL algorithms need hundreds of episodes to learn a policy. In one episode, the discounted cumulative reward get by agent is deﬁned as: G= r 0 + r 1 + 2r 2 + ; (3) Gis called return of the episode. Typical RL algorithms aim to ﬁnd an optimal policy which maxim”
Forward Modeling for Partial Observation Strategy Games - A StarCraft Defogger (2018, background): “To our best knowledge, the only deep learning approach utilizing replays of human games to improve actual full-game play is [18], which use a feed-forward model to predict the next unit that the player should produce.”
Grandmaster level in StarCraft II using multi-agent reinforcement learning (2019, unknown): “Reinforcement learning has also been studied to control units in the game 7,34,42–44 , and imitation learning has been proposed to learn unit and building compositions 45 .”
Deep Reinforcement Learning: An Overview (2017, background): “oach learned strategies for coordination similar to the level of experienced human players, like move without collision, hit and run, cover attack, and focus ﬁre without overkill.Usunier et al.(2017);Justesen and Risi (2017) also studied StarCraft. Oh et al.(2016) andTessler et al.(2017) studied Min”