On Reinforcement Learning for Full-length Game of StarCraft

By Zhen-Jia Pang, Ruo-Ze Liu, Zhou-Yu Meng, Y. Zhang, Yang Yu, Tong Lu. 2018

93 citations recorded by the source corpus.

StarCraft II poses a grand challenge for reinforcement learning. The main difficulties include huge state space, varying action space, long horizon, etc. In this paper, we investigate a set of techniques of reinforcement learning for the full-length game of StarCraft II. We investigate a hierarchical approach, where the hierarchy involves two levels of abstraction. One is the macro-actions extracted from expert’s demonstration trajectories, which can reduce the action space in an order of magnitude yet remain effective. The other is a two-layer hierarchical architecture, which is modular and easy to scale. We also investigate a curriculum transfer learning approach that trains the agent from the simplest opponent to harder ones. On a 64×64 map and using restrictive units, we train the agent on a single machine with 4 GPUs and 48 CPU threads. We achieve a winning rate of more than 99% against the difficulty level-1 built-in AI. Through the curriculum transfer learning algorithm and a mixture of combat model, we can achieve over 93% winning rate against the most difficult noncheating built-in AI (level-7) within days. We hope this study could shed some light on the future research of large-scale reinforcement learning.

Subfields: Artificial Intelligence, Sociology and Political Science

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Ahmad Beirami
2. Yunqi Zhao
3. Caedmon Somers
4. Igor Borovikov
5. Jason Rupert
6. Henrik Jacobsson

Citations in the corpus

• The StarCraft Multi-Agent Challenge (2019, neutral): “Other work focuses on playing the full game of StarCraft, including macromanagement and tactics (Pang et al., 2018; Sun et al., 2018; DeepMind, 2019).”
• On Multi-Agent Learning in Team Sports Games (2019, background): “(Pang et al., 2018) break down the complexity of the StarCraft learning environment (Vinyals et al.”
• Winning Isn't Everything: Training Human-Like Agents for Playtesting and Game AI (2019, methodology): “We create hierarchical solutions by breaking the complex problem into a hierarchy of simpler learning problems [19], [20].”