Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games

By Peng Peng, Ying Wen, Yaodong Yang, Quan Yuan, Zhenkun Tang, Haitao Long, Jun Wang. 2017

373 citations recorded by the source corpus.

Many artificial intelligence (AI) applications often require multiple intelligent agents to work in a collaborative effort. Efficient learning for intra-agent communication and coordination is an indispensable step towards general AI. In this paper, we take StarCraft combat game as a case study, where the task is to coordinate multiple agents as a team to defeat their enemies. To maintain a scalable yet effective communication protocol, we introduce a Multiagent Bidirectionally-Coordinated Network (BiCNet ['bIknet]) with a vectorised extension of actor-critic formulation. We show that BiCNet can handle different types of combats with arbitrary numbers of AI agents for both sides. Our analysis demonstrates that without any supervisions such as human demonstrations or labelled data, BiCNet could learn various types of advanced coordination strategies that have been commonly used by experienced game players. In our experiments, we evaluate our approach against multiple baselines under different scenarios; it shows state-of-the-art performance, and possesses potential values for large-scale real-world applications.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Counterfactual Multi-Agent Policy Gradients (2017, methodology): “Unlike Peng et al. (2017), our approach does not require estimating local rewards.”
Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2020, methodology): “Recent work in multi-agent RL has started moving from tabular methods (Yang and Gu, 2004; Busoniu et al., 2008) to deep learning methods that can tackle high-dimensional state and action spaces (Tampuu et al., 2017; Foerster et al., 2018; Peng et al., 2017; de Witt et al., 2018).”
QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2018, methodology): “Peng et al. (2017) also evaluate their methods on StarCraft.”
Efficient Communication in Multi-Agent Reinforcement Learning via Variance Based Control (2019, methodology): “StarCraft II [1] is a real-time strategy (RTS) game that has recently been utilized as a challenging benchmark by the reinforcement learning community [14,5,13].”
Mean Field Multi-Agent Reinforcement Learning (2018, background): “In addressing the nonstationary issue in MARL, various solutions have been proposed including neural-based opponent modeling (He & Boyd-Graber, 2016), policy parameters sharing (Gupta et al., 2017), etc. Researchers have also adopted the paradigm of centralized training with decentralized execution ”
ROMA: Multi-Agent Reinforcement Learning with Emergent Roles (2020, background): “Usunier et al. (2017) and Peng et al. (2017) study this problem from a centralized perspective.”
Learning to Schedule Communication in Multi-agent Reinforcement Learning (2019, background): “Foerster et al. (2016); Sukhbaatar et al. (2016); Peng et al. (2017); Guestrin et al. (2002), and Zhang & Lesser (2013) train multiple agents to learn a communication protocol, and have shown that communicating agents achieve better rewards at various tasks.”
TarMAC: Targeted Multi-Agent Communication (2018, background): “…networks to approximate agent policies (Hausknecht & Stone, 2015), stabilizing algorithms for multi-agent training (Lowe et al., 2017; Foerster et al., 2018), and tasks in novel domains e.g . coordination and navigation in 3D environments (Peng et al., 2017; OpenAI, 2018; Jaderberg et al., 2018).”