Episodic Exploration for Deep Deterministic Policies: An Application to StarCraft Micromanagement Tasks

By Nicolas Usunier, Gabriel Synnaeve, Zeming Lin, Soumith Chintala. 2016

142 citations recorded by the source corpus.

We consider scenarios from the real-time strategy game StarCraft as new benchmarks for reinforcement learning algorithms. We propose micromanagement tasks, which present the problem of the short-term, low-level control of army members during a battle. From a reinforcement learning point of view, these scenarios are challenging because the state-action space is very large, and because there is no obvious feature representation for the state-action evaluation function. We describe our approach to tackle the micromanagement scenarios with deep neural network controllers from raw state features given by the game engine. In addition, we present a heuristic reinforcement learning algorithm which combines direct exploration in the policy space and backpropagation. This algorithm allows for the collection of traces for learning using deterministic policies, which appears much more efficient than, for example, {\epsilon}-greedy exploration. Experiments show that with this algorithm, we successfully learn non-trivial strategies for scenarios with armies of up to 15 agents, where both Q-learning and REINFORCE struggle.

Subfields: Artificial Intelligence, Economics and Econometrics, Sociology and Political Science, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Economics and Econometrics

In Sociology and Political Science

In Reinforcement Learning

Citations in the corpus

Counterfactual Multi-Agent Policy Gradients (2017, methodology): “Previous works (Usunier et al., 2016; Peng et al., 2017) have made use of a centralised control policy that conditions on the entire state and can use powerful macro-actions, using Star-Craft’s built-in planner, that combine movement and attack actions.”
Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games (2017, methodology): “GMEZO: GreedyMDP with Episodic Zero-Order Optimisation (GMEZO) (Usunier et al. 2016) was proposed particularly to solve StarCraft micromanagement tasks.”
Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning (2017, background): “Finally, in the context of StarCraft micromanagement, Usunier et al. (2016) demonstrated some success learning a centralised policy using standard single-agent RL.”
TorchCraft: a Library for Machine Learning Research on Real-Time Strategy Games (2016, methodology): “Most previous reinforcement learning research involve simple models or limited experimental settings [26, 23].”
The StarCraft Multi-Agent Challenge (2019, methodology): “…games; work on applying deep RL algorithms to single-agent and multi-agent versions of the micromanagement tasks has thus been steadily appearing (Usunier et al., 2016; Foerster et al., 2017, 2018a; Rashid et al., 2018; Nardelli et al., 2018; Hu et al., 2018; Shao et al., 2018; Foerster et al.,…”
QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2018, methodology): “Previous work (Usunier et al., 2017; Foerster et al., 2018; Peng et al., 2017) applied RL to the original version of Star-Craft: BW, which made use of the standard API or related wrappers (Synnaeve et al., 2016).”
Multi-Agent Common Knowledge Reinforcement Learning (2018, background): “Thomas et al. (2014) explore the psychology of common knowledge and coordination.”
Evolution Strategies as a Scalable Alternative to Reinforcement Learning (2017, methodology): “An interesting hybrid of black-box optimization and policy gradient methods was recently explored by Usunier et al. [2016].”
Schema Networks: Zero-shot Transfer with a Generative Causal Model of Intuitive Physics (2017, positive): “This ﬁnding corroborates recent attempts (Usunier et al., 2016; Garnelo et al., 2016; Chang et al., 2016; Battaglia et al., 2016) to incorporate object and relational structure into neural network-based models.”
A Brief Survey of Deep Reinforcement Learning (2017, neutral): “[125] later proposed a similar method that performed”
ROMA: Multi-Agent Reinforcement Learning with Emergent Roles (2020, background): “Usunier et al. (2017) and Peng et al. (2017) study this problem from a centralized perspective.”
CESMA: Centralized Expert Supervises Multi-Agents (2019, background): “From a more centralized view point, [27, 32, 42] derived a centralized Q-value function for MARL, and in [49], they train a centralized controller and then sequentially select actions for each agent.”
Hierarchical Decision Making by Generating and Following Natural Language Instructions (2019, unknown): “Inspired by the recent success of deep reinforcement learning, more works focus on training a neural network to finish sub-tasks [24, 17], some with strong computational requirement [28].”
StarCraft Micromanagement With Reinforcement Learning and Curriculum Transfer Learning (2018, methodology): “3) GMEZO: A DRL method, based on the zero-order optimization, having impressive results over traditional RL methods [20].”