Variational Automatic Curriculum Learning for Sparse-Reward Cooperative Multi-Agent Problems
By Jiayu Chen, Yuanxing Zhang, Yuanfan Xu, Huimin Ma, Huazhong Yang, Jiaming Song, Yu Wang, Yi Wu. 2021
46 citations recorded by the source corpus.
We introduce a curriculum learning algorithm, Variational Automatic Curriculum Learning (VACL), for solving challenging goal-conditioned cooperative multi-agent reinforcement learning problems. We motivate our paradigm through a variational perspective, where the learning objective can be decomposed into two terms: task learning on the current task distribution, and curriculum update to a new task distribution. Local optimization over the second term suggests that the curriculum should gradually expand the training tasks from easy to hard. Our VACL algorithm implements this variational paradigm with two practical components, task expansion and entity progression, which produces training curricula over both the task configurations as well as the number of entities in the task. Experiment results show that VACL solves a collection of sparse-reward problems with a large number of agents. Particularly, using a single desktop machine, VACL achieves 98% coverage rate with 100 agents in the simple-spread benchmark and reproduces the ramp-use behavior originally shown in OpenAI's hide-and-seek project. Our project website is at https://sites.google.com/view/vacl-neurips-2021.
Subfields: Artificial Intelligence, Information Systems
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Matthew Riemer
- Irina Rish
- Khimya Khetarpal
- Doina Precup
- Sergey Levine
- Joelle Pineau
- Eric Eaton
- Chelsea Finn
In Information Systems
Citations in the corpus
- Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “For example, agents have in some settings learned to generate their own imagined environments or scenarios for future play (Wang et al., 2019; Raileanu et al., 2021; Chen et al., 2021; Lee and Chung, 2021), their own goals (Florensa et al.”