Inverse Reinforcement Learning in Swarm Systems

By Adrian Šošić, Wasiur R. KhudaBukhsh, Abdelhak M. Zoubir, Heinz Koeppl. 2016

82 citations recorded by the source corpus.

Inverse reinforcement learning (IRL) has become a useful tool for learning behavioral models from demonstration data. However, IRL remains mostly unexplored for multi-agent systems. In this paper, we show how the principle of IRL can be extended to homogeneous large-scale problems, inspired by the collective swarming behavior of natural systems. In particular, we make the following contributions to the field: 1) We introduce the swarMDP framework, a sub-class of decentralized partially observable Markov decision processes endowed with a swarm characterization. 2) Exploiting the inherent homogeneity of this framework, we reduce the resulting multi-agent IRL problem to a single-agent one by proving that the agent-specific value functions in this model coincide. 3) To solve the corresponding control problem, we propose a novel heterogeneous learning scheme that is particularly tailored to the swarm setting. Results on two example systems demonstrate that our framework is able to produce meaningful local reward models from which we can replicate the observed global system dynamics.

Subfields: Reinforcement Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Robotics & Control

Citations in the corpus

Learning Multi-Arm Manipulation Through Collaborative Teleoperation (2020, methodology): “Multi-Agent Imitation Learning: Most work in Multi-Agent Imitation Learning [4, 19, 32, 33] focuses on the paradigm of Inverse Reinforcement Learning [1, 2], in which multi-agent demonstrations are used to infer a reward function, and the reward function is optimized via Reinforcement Learning (RL).”
Action Semantics Network: Considering the Effects of Actions in Multiagent Systems (2019, background): “Recently, a number of deep multiagent reinforcement learning (MARL) approaches have been proposed to address complex multiagent problems, e.g., coordination of robot swarm systems (Sosic et al., 2017) and autonomous cars (Oh et al., 2015).”
Guided Deep Reinforcement Learning for Swarm Systems (2017, methodology): “We formulate the swarm system as a swarm MDP (see [16] for a similar definition) which can be seen as a special case of a decentralized partially observed Markov decision process (Dec-POMDP) [12].”
Local Communication Protocols for Learning Complex Swarm Behaviors with Deep Reinforcement Learning (2017, methodology): “We formulate the swarm system as a swarmMDP (see [24] for a similar definition) which can be seen as a special case of a decentralized”
Multi-Agent Generative Adversarial Imitation Learning (2018, background): “The most common case is fully cooperative agents, where the challenges mainly lie in other factors, such as unknown role assignments (Le et al., 2017), scalability to swarm systems (Šošic et al., 2016) and agents with partial observations (Bogert and Doshi, 2014).”