Evolutionary Population Curriculum for Scaling Multi-Agent Reinforcement Learning

By Qian Long, Zihan Zhou, Abhinav Kumar Gupta, Fei Fang, Yi Wu, Xiaolong Wang. 2020

96 citations recorded by the source corpus.

In multi-agent games, the complexity of the environment can grow exponentially as the number of agents increases, so it is particularly challenging to learn good policies when the agent population is large. In this paper, we introduce Evolutionary Population Curriculum (EPC), a curriculum learning paradigm that scales up Multi-Agent Reinforcement Learning (MARL) by progressively increasing the population of training agents in a stage-wise manner. Furthermore, EPC uses an evolutionary approach to fix an objective misalignment issue throughout the curriculum: agents successfully trained in an early stage with a small population are not necessarily the best candidates for adapting to later stages with scaled populations. Concretely, EPC maintains multiple sets of agents in each stage, performs mix-and-match and fine-tuning over these sets and promotes the sets of agents with the best adaptability to the next stage. We implement EPC on a popular MARL algorithm, MADDPG, and empirically show that our approach consistently outperforms baselines by a large margin as the number of agents grows exponentially.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomization (2021, methodology): “We released the environment to the community as a novel testbed for MARL research.”
Learning scheduling algorithms for data processing clusters (2018, background): “To make the RL problem tractable, one potential research direction is to leverage multi-agent RL [38, 50, 62].”
Modeling the Interaction between Agents in Cooperative Multi-Agent Reinforcement Learning (2021, methodology): “When directly applying the existing MARL algorithms to complex games with a large number of agents, agents may fail to learn good strategies and end up with little interaction with other agents even when collaboration is significantly beneficial [19].”
Randomized Entity-wise Factorization for Multi-Agent Reinforcement Learning (2020, background): “This paper focuses on cooperative MARL with centralized training and decentralized execution (Oliehoek et al., 2016, CTDE).”
Multi-agent deep reinforcement learning: a survey (2021, methodology): “Evolutionary algorithms have been used in versatile contexts of multi-agent RL, e.g. for building intrinsic motivation (Wang et al. 2019), shaping rewards (Jaderberg et al. 2019), generating curricula (Long et al. 2020) and analyzing dynamics (Bloembergen et al. 2015).”
Variational Automatic Curriculum Learning for Sparse-Reward Cooperative Multi-Agent Problems (2021, background): “[18, 31] propose population curriculum, which trains entity-invariant policies over training tasks with a growing number of agents.”