CM3: Cooperative Multi-goal Multi-stage Multi-agent Reinforcement Learning

By Jiachen Yang, Alireza Nakhaei, David Isele, Hongyuan Zha, Kikuo Fujimura. 2018

87 citations recorded by the source corpus.

A variety of cooperative multi-agent control problems require agents to achieve individual goals while contributing to collective success. This multi-goal multi-agent setting poses difficulties for recent algorithms, which primarily target settings with a single global reward, due to two new challenges: efficient exploration for learning both individual goal attainment and cooperation for others' success, and credit-assignment for interactions between actions and goals of different agents. To address both challenges, we restructure the problem into a novel two-stage curriculum, in which single-agent goal attainment is learned prior to learning multi-agent cooperation, and we derive a new multi-goal multi-agent policy gradient with a credit function for localized credit assignment. We use a function augmentation scheme to bridge value and policy functions across the curriculum. The complete architecture, called CM3, learns significantly faster than direct adaptations of existing algorithms on three challenging multi-goal multi-agent problems: cooperative navigation in difficult formations, negotiating multi-vehicle lane changes in the SUMO traffic simulator, and strategic cooperation in a Checkers environment.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Computer Networks and Communications

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Computer Networks and Communications

Citations in the corpus

Hierarchical Cooperative Multi-Agent Reinforcement Learning with Skill Discovery (2019, methodology): “choose from a diverse set of cooperative MARL algorithms with decentralized execution [10, 32, 36, 44].”
Integrating Independent and Centralized Multi-agent Reinforcement Learning for Traffic Signal Network Optimization (2019, background): “While IQL and IAC agents display strong ability to optimize individual rewards [29, 33], the lack of global information and a mechanism for cooperation means they are likely to settle for sub-optimal solutions.”
Multi-agent deep reinforcement learning: a survey (2021, background): “However, global optimality can only be accomplished if agents also allow others to be successful in their task (Yang et al. 2020).”
On Multi-Agent Learning in Team Sports Games (2019, result): “Similar approaches are reported in (Yang et al., 2018) to achieve cooperation in simulating self-driving cars, and in (Gao et al.”
A review of cooperative multi-agent deep reinforcement learning (2019, background): “[156] considers a multi-agent cooperative problem in which in addition to the cooperative goal, each agent needs to attain some personal goals.”
Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey (2020, background): “For example, Yang et al. (2020) propose CM3, which takes a two-stage approach.”