Multi-Agent Generative Adversarial Imitation Learning

By Jiaming Song, Hongyu Ren, Dorsa Sadigh, Stefano Ermon. 2018

263 citations recorded by the source corpus.

Imitation learning algorithms can be used to learn a policy from expert demonstrations without access to a reward signal. However, most existing approaches are not applicable in multi-agent settings due to the existence of multiple (Nash) equilibria and non-stationary environments. We propose a new framework for multi-agent imitation learning for general Markov games, where we build upon a generalized notion of inverse reinforcement learning. We further introduce a practical multi-agent actor-critic algorithm with good empirical performance. Our method can be used to imitate complex behaviors in high-dimensional environments with multiple cooperative or competing agents.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Learning Multi-Arm Manipulation Through Collaborative Teleoperation (2020, methodology): “While centralized and decentralized structures for policies and reward functions have been studied extensively in the multi-agent IRL setting [35], they have not been explored significantly in BC settings.”
TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrations (2021, positive): “And then Song et al. (2018) extended GAIL to multi-agent scenarios.”
Constructing Unrestricted Adversarial Examples with Generative Models (2018, background): “Generative adversarial networks (GANs) A GAN [19, 20, 34, 35, 36] is composed of a generator gθ(z) and a discriminator dφ(x).”
Deep Reinforcement Learning (2018, methodology): “ould control low level actions. The authors experiment on autonomous highway driving using TORCS driving simulator (Wymann et al., 2014). See the open source at https://github.com/ermongroup/InfoGAIL.Song et al. (2018) extend GAIL to the multi-agent setting. REWARD MANIPULATING van Seijen et al.(201”
Multi-task Maximum Entropy Inverse Reinforcement Learning (2018, methodology): “Recent extensions to GAIL augment trajectories with a latent intention variable that speciﬁes the task, and then maximise mutual information between the state-action pairs and intention variable (Hausman et al., 2017; Li et al., 2017).”
Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gap (2021, background): “Starting with the seminal work of (Ho and Ermon 2016), numerous proposed approaches have framed imitation learning as a game be-tween a learner’s policy and another network that attempts to discriminate between learner rollouts and expert demonstrations (Fu et al. 2018, Song et al. 2018).”
Multi-objective multi-agent decision making: a utility-based analysis and survey (2019, background): “The paradigmof centralised trainingwith decentralised execution represents a middle ground between fully centralised and decentralised settings often used in cooperative or mixed settings [53,54,81,86,112,144,156].”
Multi-agent deep reinforcement learning: a survey (2021, background): “2017) and Markov Games with mixed settings (Song et al. 2018).”
Generating Multi-Agent Trajectories using Programmatic Weak Supervision (2018, background): “There has been some prior work in multi-agent imitation learning (Le et al., 2017; Song et al., 2018) and learning stochastic policies (Ho & Ermon, 2016; Li et al., 2017), but no previous work has focused on learning generative polices while simultaneously addressing generative and multi-agent…”
A Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges (2023, unknown): “AIL demonstrates signiﬁcant enhancements over the existing methods across various benchmark environments, including robotics, autonomous driving, and gaming [45], [74], [75].”
Imitation Learning: A Survey of Learning Methods, Environments and Metrics (2024, unknown): “In Song et al.’s work [53], the method is divided into centralised (or cooperative), decentralised, and zero-sum.”
A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications (2020, background): “GANs’ Representative variants InfoGAN [14], cGANs [15], CycleGAN [16], f -GAN [17], WGAN [18], WGAN-GP [19], LS-GAN [20] Objective function LSGANs [21], [22], hinge loss based GAN [23]–[25], MDGAN [26], unrolled GAN [27], SN-GANs [23], RGANs [28] Skills ImprovedGANs [29], AC-GAN [30] LAPGAN [31], DC”
Learning From Demonstration in the Wild (2018, background): “In fact, pretraining GAIL with BC is known to be beneficial [33], [30], [48], [49].”