Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?

By Christian Schröder de Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip H. S. Torr, Mingfei Sun, Shimon Whiteson. 2020

584 citations recorded by the source corpus.

Most recently developed approaches to cooperative multi-agent reinforcement learning in the \emph{centralized training with decentralized execution} setting involve estimating a centralized, joint value function. In this paper, we demonstrate that, despite its various theoretical shortcomings, Independent PPO (IPPO), a form of independent learning in which each agent simply estimates its local value function, can perform just as well as or better than state-of-the-art joint learning approaches on popular multi-agent benchmark suite SMAC with little hyperparameter tuning. We also compare IPPO to several variants; the results suggest that IPPO's strong performance may be due to its robustness to some forms of environment non-stationarity.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yaodong Yang
2. Jun Wang
3. Ying Wen
4. Hongyuan Zha
5. Yu Wang
6. Chao Yu
7. Alexandre M. Bayen
8. Eugene Vinitsky

In Reinforcement Learning

1. Yaodong Yang
2. Jun Wang
3. Yu Wang
4. Alexandre M. Bayen
5. Chao Yu
6. Ying Wen
7. Eugene Vinitsky
8. Hongyuan Zha

Citations in the corpus

• The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games (2021, neutral): “[7] empirically show that decentralized, independent PPO (IPPO) can achieve high success rates in several hard SMAC maps – however, the reported IPPO results remain overall worse than QMix, and the study is limited to SMAC.”
• MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning (2021, methodology): “We benchmark several MARL algorithms: Independent policy optimization (IPPO) method [43] which uses PPO [45] as the individual learners and two centralized critic methods which encode the nearby agents’ states into the input of value functions (centralized critics).”
• Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning (2021, methodology): “We evaluate HATRPO and HAPPO on benchmarks of StarCraftII and Multi-Agent MuJoCo against strong baselines such as MADDPG (Lowe et al., 2017a), IPPO (de Witt et al., 2020b) and MAPPO (Yu et al., 2021); results clearly demonstrate its state-of-the-art performance across all tested tasks.”
• The Surprising Effectiveness of MAPPO in Cooperative, Multi-Agent Games (2021, background): “Note that although de Witt et al. (2020) empirically notice that fully decentralized independent PPO training (IPPO) can achieve surprisingly high success rates on some specific hard SMAC maps, the cause for this success is unknown, and the over-all performance of IPPO remains much worse than QMix.”
• Dealing with Non-Stationarity in MARL via Trust-Region Decomposition (2021, background): “For example, IPPO (de Witt et al., 2020) assume homogeneity of action spaces for all agents and enforce parameter sharing which is discussed above.”