The StarCraft Multi-Agent Challenge
By Mikayel Samvelyan, Tabish Rashid, Christian Schröder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philip H. S. Torr, Jakob N. Foerster, Shimon Whiteson. 2019
1,262 citations recorded by the source corpus.
In the last few years, deep multi-agent reinforcement learning (RL) has become a highly active area of research. A particularly challenging class of problems in this area is partially observable, cooperative, multi-agent learning, in which teams of agents must learn to coordinate their behaviour while conditioning only on their private observations. This is an attractive research area since such problems are relevant to a large number of real-world systems and are also more amenable to evaluation than general-sum problems. Standardised environments such as the ALE and MuJoCo have allowed single-agent RL to move beyond toy domains, such as grid worlds. However, there is no comparable benchmark for cooperative multi-agent RL. As a result, most papers in this field use one-off toy problems, making it difficult to measure real progress. In this paper, we propose the StarCraft Multi-Agent Challenge (SMAC) as a benchmark problem to fill this gap. SMAC is based on the popular real-time strategy game StarCraft II and focuses on micromanagement challenges where each unit is controlled by an independent agent that must act based on local observations. We offer a diverse set of challenge maps and recommendations for best practices in benchmarking and evaluations. We also open-source a deep multi-agent RL learning framework including state-of-the-art algorithms. We believe that SMAC can provide a standard benchmark environment for years to come. Videos of our best agents for several SMAC scenarios are available at: https://youtu.be/VZ7zmQ_obZ0.
Subfields: Artificial Intelligence, Management Science and Operations Research, LLM Agents & Tool Use, Reinforcement Learning, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Yoshua Bengio
- Mohit Bansal
- Shimon Whiteson
- Quoc V. Le
- Ruslan Salakhutdinov
- Pieter Abbeel
- Jianfeng Gao
In Management Science and Operations Research
- Shimon Whiteson
- Filippos Christianos
- Stefano V. Albrecht
- Chongjie Zhang
- Wendelin Böhmer
- Georgios Papoudakis
- Tabish Rashid
- Christian Schröder de Witt
In LLM Agents & Tool Use
- Shimon Whiteson
- Wendelin Böhmer
- Chongjie Zhang
- Tabish Rashid
- Davood Hajinezhad
- Afshin Oroojlooyjadid
- Tonghan Wang
- Stefano V. Albrecht
In Reinforcement Learning
- Shimon Whiteson
- Sergey Levine
- Pieter Abbeel
- Chongjie Zhang
- Jakob N. Foerster
- Yuxi Li
- Tonghan Wang
- Christian Schröder de Witt
In Evaluation & Benchmarks
- Shimon Whiteson
- Bolei Zhou
- Antonio Torralba
- Wendelin Böhmer
- Sanja Fidler
- Chongjie Zhang
- Tabish Rashid
- Josh Tenenbaum
Citations in the corpus
- The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games (2021, methodology): “Similar conclusions have been drawn in multi-agent domains: [24] report that multi-agent PG methods such as COMA are outperformed by MADDPG and QMix [25] by a clear margin in the particle-world environment [22] and the StarCraft multi-agent challenge [26].”
- Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2020, background): “This results in significant performance gains on some scenarios in SMAC (Samvelyan et al., 2019).”
- Counterfactual Multi-Agent Reinforcement Learning with Graph Convolution Communication (2020, neutral): “To understand the effects of the simulation’s randomness on the algorithms’ learning and performance, we followed the evaluation procedure described in [18]: the training is paused every 10 , 000 steps during which 96 test episodes are run with the latest model performing greedy action selection.”
- Celebrating Diversity in Shared Multi-Agent Reinforcement Learning (2021, methodology): “So we tune hyperparameters for SMAC both 6h _ vs _ 8z and Corridor .”
- Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharing (2021, background): “A finer grid search on the entropy coefficient for the SMAC task was performed.”
- Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge? (2020, methodology): “In this paper, we present empirical evidence for Independent PPO (IPPO), a multi-agent variant of proximal policy optimization [23], that shows IPPO matches or outperforms state-of-the-art MARL CTDE algorithms such as QMIX [19] or MAVEN [15] on multiple hard maps on SMAC [20].”
- QPLEX: Duplex Dueling Multi-Agent Q-Learning (2020, methodology): “1 TRAINING WITH ONLINE DATA COLLECTION We evaluate QPLEX in 17 benchmark tasks of StarCraft II, which contains 14 popular tasks proposed by SMAC (Samvelyan et al., 2019) and three new super hard cooperative tasks.”
- Weighted QMIX: Expanding Monotonic Value Function Factorisation (2020, methodology): “05 over 1mil timesteps, increased from the 50 k used in [26].”
- Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks (2020, methodology): “EPyMARL is an extension of the PyMARL codebase [Samvelyan et al., 2019].”
- QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2018, background): “This results in significant performance gains on some scenarios in SMAC (Samvelyan et al., 2019).”
- MAVEN: Multi-Agent Variational Exploration (2019, methodology): “Our evaluation procedure is similar to [40, 43].”
- Learning Nearly Decomposable Value Functions Via Communication Minimization (2019, methodology): “Such meth-ods has shown their effectiveness on complex task (Samvelyan et al., 2019b).”
- Deep Coordination Graphs (2019, methodology): “All algorithms are implemented in the multi-agent framework PYMARL (Samvelyan et al., 2019).”
- From Few to More: Large-scale Dynamic Multiagent Curriculum Learning (2019, methodology): “Experimental results in Starcraft-II (Samvelyan et al. 2019) and MAgent (Zheng et al.”
- Efficient Communication in Multi-Agent Reinforcement Learning via Variance Based Control (2019, methodology): “We evaluated the performance of VBC with the StarCraft Multi-Agent Challenge (SMAC) [15].”
- Action Semantics Network: Considering the Effects of Actions in Multiagent Systems (2019, methodology): “Our test domains include StarCraft II micromanagement (Samvelyan et al., 2019) and Massively Multiplayer Online Role-Playing Games (Neural MMO) (Suarez et al., 2019).”
- A survey and critique of multiagent deep reinforcement learning (2018, methodology): “oplayer two-action game and then with a more complex variant; similarly, QMIX [181] presented its results in a two-step (matrix) game and then in the more involved Starcraft II micromanagement domain [303]. 4.5. Open questions Finally, here we present some open questions for MDRL and point to sugges”