Coordinated Multi-Agent Imitation Learning
By Hoang Minh Le, Yisong Yue, Peter Carr, Patrick Lucey. 2017
212 citations recorded by the source corpus.
We study the problem of imitation learning from demonstrations of multiple coordinating agents. One key challenge in this setting is that learning a good model of coordination can be difficult, since coordination is often implicit in the demonstrations and must be inferred as a latent variable. We propose a joint approach that simultaneously learns a latent coordination model along with the individual policies. In particular, our method integrates unsupervised structure learning with conventional imitation learning. We illustrate the power of our approach on a difficult problem of learning multiple policies for finegrained behavior modeling in team sports, where different players occupy different roles in the coordinated team strategy. We show that having a coordination model to infer the roles of players yields substantially improved imitation loss compared to conventional baselines.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Economics and Econometrics, LLM Agents & Tool Use, Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Pieter Abbeel
- Li Fei-Fei
- Chelsea Finn
- Yoshua Bengio
- Ruslan Salakhutdinov
- Mohit Bansal
- Trevor Darrell
In Computer Vision and Pattern Recognition
- Dacheng Tao
- Luc Van Gool
- Trevor Darrell
- Li Fei-Fei
- Kristen Grauman
- Jitendra Malik
- Xiaogang Wang
- Ming-Hsuan Yang
In Economics and Econometrics
- Zongqing Lu
- Jiechuan Jiang
- Bangwei Li
- Yongyuan Liang
- Yi Cai
- Hongyuan Zha
- Klaus-Jürgen Diepold
- Silvio Savarese
In LLM Agents & Tool Use
- Bangwei Li
- Yongyuan Liang
- Maruan Al-Shedivat
- Harrison Edwards
- Hongyuan Zha
- Klaus-Jürgen Diepold
- Jonathan P. How
- Sergey Levine
In Reinforcement Learning
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Silvio Savarese
- Li Fei-Fei
- Tengyu Ma
- Yuke Zhu
- Roberto Mart'in-Mart'in
Citations in the corpus
- Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling (2019, methodology): “…adopted as a standard approach in robotics [Pomerleau, 1989, Schaal, 1997, Argall et al., 2009, Osa et al., 2017, Ye and Alterovitz, 2017, Aleotti and Caselli, 2006, Lawitzky et al., 2012, Torabi et al., 2018, Le et al., 2017, 2018] and many other areas such as playing games [Mnih et al., 2013].”
- Learning Multi-Arm Manipulation Through Collaborative Teleoperation (2020, methodology): “Multi-Agent Imitation Learning: Most work in MultiAgent Imitation Learning [4, 20, 35, 36] focuses on the paradigm of Inverse Reinforcement Learning [1, 2], in which multi-agent demonstrations are used to infer a reward function, and the reward function is optimized via Reinforcement”
- Hierarchical Cooperative Multi-Agent Reinforcement Learning with Skill Discovery (2019, background): “intuition is that a skill in many complex fully-cooperative team games can be inferred from the trajectory generated by a player who performs primitive actions that implement the skill [18, 14].”
- Learning Attentional Communication for Multi-Agent Cooperation (2018, background): “However, MADDPG has to train an independent policy network for each agent, where each agent would learn a policy specializing specific tasks [10], and the policy network easily overfits to the number of agents.”
- UvA-DARE (Digital Academic Repository) Neural Relational Inference for Interacting Systems Neural Relational Inference for Interacting Systems (2018, unknown): “Recent related works on graph-based methods for human motion prediction include (Alahi et al., 2016) where the graph is not learned but is based on proximity and (Le et al., 2017) tries to cluster agents into roles.”
- Learning Hierarchical Teaching Policies for Cooperative Agents (2019, background): “Recent work applied imitation learning for multiagent coordination [19] and explored effective combinations of imitation learning and HRL [18].”
- Parallel Knowledge Transfer in Multi-Agent Reinforcement Learning (2020, methodology): “And Knowledge transfer method has been studied in several related fields including imitation learning, learning from demonstration [4], and inverse reinforcement”
- PRECOG: PREdiction Conditioned on Goals in Visual Multi-Agent Settings (2019, background): “Multi-agent forecasting: Data-driven approaches have been applied to forecast complex interactions between multiple pedestrians [1, 3, 10, 14, 21], vehicles [6, 19, 26], and athletes [9, 18, 20, 34, 36, 37].”
- Generative Multi-Agent Behavioral Cloning (2018, methodology): “In principle, we can provide the defensive positions as conditional input for our model and update the defensive positions using methods such as Le et al. [2017]. We also ignore the ball since the ball dynamics are difficult to learn (e.”
- Learning to Teach in Cooperative Multiagent Reinforcement Learning (2018, unknown): “These include imitation learning of expert demonstrations (Le et al. 2017), cooperative inverse reinforcement learning with a human and robot (Hadfield-Menell et al. 2016), and transfer to parallel learners in tasks with similar value functions (Taylor et al. 2013).”
- Learning Policy Representations in Multiagent Systems (2018, background): “Imitation learning has been extensively studied in the single-agent setting and recent work by Le et al. (2017) proposes an algorithm for imitation in a coordinated multiagent system.”
- Generating Multi-Agent Trajectories using Programmatic Weak Supervision (2018, methodology): “There has been some prior work in multi-agent imitation learning (Le et al., 2017; Song et al., 2018) and learning stochastic policies (Ho & Ermon, 2016; Li et al.”
- A Q-values Sharing Framework for Multi-agent Reinforcement Learning under Budget Constraint (2020, background): “Therefore, it has been extensively used to evaluate multiagent learning algorithms [35, 36, 37].”
- Multi-agent deep reinforcement learning: a survey (2021, background): “…models can be found in cooperative tasks (Barde et al. 2019; Tacchetti et al. 2019; Zheng et al. 2018b) with imitation learning (Grover et al. 2018; Le et al. 2017), in social dilemmas (Jaques et al. 2019; Letcher et al. 2019), and by predicting behaviors from observations (Hong et al. 2017;…”