ROMA: Multi-Agent Reinforcement Learning with Emergent Roles

By Tonghan Wang, Heng Dong, Victor R. Lesser, Chongjie Zhang. 2020

304 citations recorded by the source corpus.

The role concept provides a useful tool to design and understand complex multi-agent systems, which allows agents with a similar role to share similar behaviors. However, existing role-based methods use prior domain knowledge and predefine role structures and behaviors. In contrast, multi-agent reinforcement learning (MARL) provides flexibility and adaptability, but less efficiency in complex tasks. In this paper, we synergize these two paradigms and propose a role-oriented MARL framework (ROMA). In this framework, roles are emergent, and agents with similar roles tend to share their learning and to be specialized on certain sub-tasks. To this end, we construct a stochastic role embedding space by introducing two novel regularizers and conditioning individual policies on roles. Experiments show that our method can learn specialized, dynamic, and identifiable roles, which help our method push forward the state of the art on the StarCraft II micromanagement benchmark. Demonstrative videos are available at this https URL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Celebrating Diversity in Shared Multi-Agent Reinforcement Learning (2021, background): “While parameter sharing has been proven to accelerate training [17], its drawbacks are also apparent in complex tasks.”
Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharing (2021, background): “ROMA (Wang et al., 2020) learns dynamic roles to share experience between agents that perform similar tasks.”
RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, methodology): “However, learning roles from scratch might not be easier than learning without roles, as directly ﬁnding an optimal decomposition suffers from the same problem as other CTDE learning methods – searching in the large joint space with substantial exploration (Wang et al., 2020c).”
Human-Timescale Adaptation in an Open-Ended Task Space (2023, background): “Recent works have demonstrated the effectiveness of agent-environment co-adaptation with a distribution of tasks (Wang et al., 2019; Akkaya et al., 2019; Wang et al., 2020a; Parker-Holder et al., 2022).”
Shared Experience Actor-Critic for Multi-Agent Reinforcement Learning (2020, methodology): “In Table 1 we also present the ﬁnal evaluation returns of three state-of-the-art MARL algorithms (QMIX [28], MADDPG [20], and ROMA [36]) on a selection of tasks.”
Towards Understanding Linear Value Decomposition in Cooperative Multi-Agent Q-Learning (2020, background): “[18] Tonghan Wang, Heng Dong, Victor Lesser, and Chongjie Zhang.”
Off-Policy Multi-Agent Decomposed Policy Gradients (2020, methodology): “Cooperative multi-agent reinforcement learning (MARL) has achieved great progress in recent years (Hughes et al., 2018; Jaques et al., 2019; Vinyals et al., 2019; Zhang et al., 2019; Baker et al., 2020; Wang et al., 2020c).”
The Emergence of Individuality in Multi-Agent Reinforcement Learning (2020, methodology): “However, ROMA converges to a lower mean reward than EOI+QMIX and learns much slower.”
QTRAN++: Improved Value Transformation for Cooperative Multi-Agent Reinforcement Learning (2020, methodology): “Recently, ROMA (Wang et al., 2020b) proposes a role-oriented framework to learn roles via deep RL with regularizers and role-conditioned policies.”
Multi-agent deep reinforcement learning: a survey (2021, background): “…by applying intrinsic motivation (Jaques et al. 2018, 2019), by making the communication protocol more robust or scalable (Kim et al. 2019; Singh et al. 2019), and maximizing the utility of the communication through efficient encoding (Celikyilmaz et al. 2018; Li et al. 2019b; Wang et al. 2020c).”