Learning multi-agent state space representations

By Yann-Michaël De Hauwere, Peter Vrancx, Ann Nowé. 2010

74 citations recorded by the source corpus.

This paper describes an algorithm, called CQ-learning, which learns to adapt the state representation for multi-agent systems in order to coordinate with other agents. We propose a multi-level approach which builds a progressively more advanced representation of the learning problem. The idea is that agents start with a minimal single agent state space representation, which is expanded only when necessary. In cases where agents detect conflicts, they automatically expand their state to explicitly take into account the other agents. These conflict situations are then analyzed in an attempt to find an abstract representation which generalises over the problem states. Our system allows agents to learn effective policies, while avoiding the exponential state space growth typical in multi-agent environments. Furthermore, the method we introduce to generalise over conflict states allows knowledge to be transferred to unseen and possibly more complex situations. Our research departs from previous efforts in this area of multi-agent learning because our agents combine state space generalisation with an agent-centric point of view. The algorithms that we introduce can be used in robotic systems to automatically reduce the sensor information to what is essential to solve the problem at hand. This is a must when dealing with multiple agents, since learning in such environments is a cumbersome task due to the massive amount of information, much of which may be irrelevant. In our experiments we demonstrate a simulation of such environments using various gridworlds.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Value Function Transfer for Deep Multi-Agent Reinforcement Learning Based on N-Step Returns (2019, methodology): “A straightforward solution is the immediate rewards in each state, which is adopted in CQ-learning algorithm [De Hauwere et al. , 2010].”
A survey and critique of multiagent deep reinforcement learning (2018, background): “To learn complex multiagent interactions some type of abstraction [84] is often needed, for example, factored value functions [8,119–121,174,236] (see QMIX and VDN in Sect.”
Context-sensitive reward shaping for sparse interaction multi-agent systems (2016, methodology): “De Hauwere et al . introduced CQ-learning for solving a DEC-SIMDP (de Hauwere et al ., 2010).”
A Review of Safe Reinforcement Learning: Methods, Theory and Applications (2022, methodology): “Specifically, an attempt is made to ensure safety of multi-agent systems by leveraging central shielding or factored shielding methods, which are used to specify the safe actions that agents only can take, and also correct the unsafe actions that agents explored, furthermore, they evaluate the metho”
A Q-values Sharing Framework for Multi-agent Reinforcement Learning under Budget Constraint (2020, unknown): “al propose CQ-learning [15] and FCQ-learning [16].”
Multiagent Learning of Coordination in Loosely Coupled Multiagent Systems (2015, background): “The decentralized decision makings in Dec-MDPs thus bear an extremely high complexity (i.e., NEXP-complete) to achieve a globally optimal policy, even in the two-agent case, versus the P-completeness of a single MDP and multiagent MDPs [28].”
A Survey of Learning in Multiagent Environments: Dealing with Non-Stationarity (2017, background): “A different approach is to determine the degree of interaction among agents, this could help in defining whether to interact with an agent or ignore it and take it as part of the environment (De Hauwere et al., 2010; Yu et al., 2015).”
Learning in Multi-agent Systems with Sparse Interactions by Knowledge Transfer and Game Abstraction (2015, methodology): “Note that a similar problem has been investigated in CQlearning [4] and FCQ-learning [9].”