Guided Deep Reinforcement Learning for Swarm Systems

By Maximilian Hüttenrauch, Adrian Šošić, Gerhard Neumann. 2017

149 citations recorded by the source corpus.

In this paper, we investigate how to learn to control a group of cooperative agents with limited sensing capabilities such as robot swarms. The agents have only very basic sensor capabilities, yet in a group they can accomplish sophisticated tasks, such as distributed assembly or search and rescue tasks. Learning a policy for a group of agents is difficult due to distributed partial observability of the state. Here, we follow a guided approach where a critic has central access to the global state during learning, which simplifies the policy evaluation problem from a reinforcement learning point of view. For example, we can get the positions of all robots of the swarm using a camera image of a scene. This camera image is only available to the critic and not to the control policies of the robots. We follow an actor-critic approach, where the actors base their decisions only on locally sensed information. In contrast, the critic is learned based on the true global state. Our algorithm uses deep reinforcement learning to approximate both the Q-function and the policy. The performance of the algorithm is evaluated on two tasks with simple simulated 2D agents: 1) finding and maintaining a certain distance to each others and 2) locating a target.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Computer Networks and Communications, Reinforcement Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Computer Networks and Communications

In Reinforcement Learning

In Robotics & Control

Citations in the corpus

Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2020, background): “Reinforcement learning (RL) holds considerable promise to help address a variety of cooperative multi-agent problems, such as coordination of robot swarms (H¨uttenrauch et al., 2017) and autonomous cars (Cao et al., 2012).”
Celebrating Diversity in Shared Multi-Agent Reinforcement Learning (2021, background): “Cooperative multi-agent reinforcement learning (MARL) has drawn increasing interest in recent years, which provides a promise for solving many real-world challenging problems, such as sensor networks [1], traffic management [2], and coordination of robot swarms [3].”
Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge? (2020, background): “Many practical control problems feature a team of multiple agents that must coordinate to achieve a common goal [5, 12].”
QPLEX: Duplex Dueling Multi-Agent Q-Learning (2020, background): “Cooperative multi-agent reinforcement learning (MARL) has broad prospects for addressing many complex real-world problems, such as sensor networks (Zhang & Lesser, 2011), coordination of robot swarms (Hüttenrauch et al., 2017), and autonomous cars (Cao et al., 2012).”
QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2018, background): “Reinforcement learning (RL) holds considerable promise to help address a variety of cooperative multi-agent problems, such as coordination of robot swarms (Hüttenrauch et al., 2017) and autonomous cars (Cao et al.”
MAVEN: Multi-Agent Variational Exploration (2019, methodology): “Cooperative multi-agent reinforcement learning (MARL) is a key tool for addressing many real-world problems such as coordination of robot swarms [22] and autonomous cars [6].”
Towards Understanding Linear Value Decomposition in Cooperative Multi-Agent Q-Learning (2020, background): “Cooperative multi-agent reinforcement learning (MARL) has great promise for addressing coordination problems in a variety of applications, such as robotic systems [1], autonomous cars [2], and sensor networks [3].”
Deep Reinforcement Learning for Multiagent Systems: A Review of Challenges, Solutions, and Applications (2018, methodology): “The swarm systems are formulated in [100] as a special case of the decentralized POMDP [101] and used an AC deep RL approach to control a group of cooperative agents.”