Learning of coordination: exploiting sparse interactions in multiagent systems

By Francisco S. Melo, Manuela M. Veloso. 2009

89 citations recorded by the source corpus.

Creating coordinated multiagent policies in environments with uncertainty is a challenging problem, which can be greatly simplified if the coordination needs are known to be limited to specific parts of the state space, as previous work has successfully shown. In this work, we assume that such needs are unknown and we investigate coordination learning in multiagent settings. We contribute a reinforcement learning based algorithm in which independent decision-makers/agents learn both individual policies and when and how to coordinate. We focus on problems in which the interaction between the agents is sparse, exploiting this property to minimize the coupling of the learning processes for the different agents. We introduce a two-layer extension of Q-learning, in which we augment the action space of each agent with a coordination action that uses information from other agents to decide the correct action. Our results show that our agents learn both to act coordinate and to act independently, in the different regions of the space where they need to, and need not to, coordinate, respectively.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Y. Liu
2. Yujing Hu
3. Yang Gao
4. Yann-Michaël De Hauwere
5. Sam Devlin
6. Ann Nowé
7. Yingfeng Chen
8. Daniel Kudenko

Citations in the corpus

• Value Function Transfer for Deep Multi-Agent Reinforcement Learning Based on N-Step Returns (2019, methodology): “Melo and Veloso present examples of the so-called multi-agent systems with sparse interactions [Melo and Veloso, 2009].”
• Learning multi-agent state space representations (2010, unknown): “This is a significant difference to previous work [12], where the avoidance of coordination in such a situation is highly dependent on the penalty for miscoordination in that situation and how often it occurs.”
• Context-sensitive reward shaping for sparse interaction multi-agent systems (2016, methodology): “Note that there is a clear distinction between being able to observe the other agents and learn when to use this information, versus always learning using this information as illustrated in Melo and Veloso (2009) and de Hauwere et al .”
• Learning in Multi-agent Systems with Sparse Interactions by Knowledge Transfer and Game Abstraction (2015, methodology): “This motivates the more recent work that learns the interaction areas during the learning process [4,9,13,17,18].”
• Game Theory and Multi-agent Reinforcement Learning (2012, background): “In later work, Melo and Veloso introduced an algorithm where agents learn in which states they need to condition their actions on the local state information of other agents (Melo and Veloso, 2009).”
• Multiagent Learning of Coordination in Loosely Coupled Multiagent Systems (2015, background): “[47] proposed a two-layer extension of the Q-learning algorithm to enable agents to learn where coordination is be neficial by augmenting the action space with a pseudo-coordination a ction.”