Self-organization for coordinating decentralized reinforcement learning

By Chongjie Zhang, Victor R. Lesser, Sherief Abdallah. 2010

42 citations recorded by the source corpus.

Decentralized reinforcement learning (DRL) has been applied to a number of distributed applications. However, one of the main challenges faced by DRL is its convergence. Previous work has shown that hierarchically organizational control is an effective way of coordinating DRL to improve its speed, quality, and likelihood of convergence. In this paper, we develop a distributed, negotiation-based approach to dynamically forming such hierarchical organizations. To reduce the complexity of coordinating DRL, our self-organization approach groups strongly-interacting learning agents together, whose exploration strategies are coordinated by one supervisor. We formalize this idea by characterizing interactions among agents in a decentralized Markov Decision Process model and defining and analyzing a measure that explicitly captures the strength of such interactions. Experimental results show that our dynamically evolving organizations outperform predefined organizations for coordinating DRL.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, background): “In multi-agent settings, many challenges, such as efﬁcient communication (Ossenkopf et al., 2019) and labor division (Wang et al., 2020c) in large systems, necessitate hierarchical learning along the second dimension – over agents (Zhang et al., 2010).”
Coordinated Multi-Agent Reinforcement Learning in Networked Distributed POMDPs (2011, neutral): “Our previous work (Zhang, Abdallah, and Lesser 2009; Zhang, Lesser, and Abdallah 2010) presented a general supervisory framework for coordinating MARL, but did not provide a general coordination algorithm.”
Coordinating multi-agent reinforcement learning with limited communication (2013, background): “Note that the self-organization approach for coordinating MARL in [15] is also a co-adaptation approach, which allows agents to dynamically evolving supervisory organizations for MASPA [14] to better coordinate their learning processes while they are concurrently learning their operational policies.”
Multiagent Learning of Coordination in Loosely Coupled Multiagent Systems (2015, background): “Multi-Agent Learning (MAL) provides a promising paradigm f or studying how agents can learn coordinated behaviors in Mult i-Agent Systems (MASs) [1], and it is finding increasing application s i a wide range of real-world domains such as robotics [2], [3], d istributed control [4], [5], resource man”
A Concise Introduction to Decentralized POMDPs (2016, methodology): “This approach was extended to a hierarchical approach that includes so-called supervisors [Zhang et al., 2010].”