Learning Policy Representations in Multiagent Systems

By Aditya Grover, Maruan Al-Shedivat, Jayesh K. Gupta, Yuri Burda, Harrison Edwards. 2018

137 citations recorded by the source corpus.

Modeling agent behavior is central to understanding the emergence of complex phenomena in multiagent systems. Prior work in agent modeling has largely been task-specific and driven by hand-engineering domain-specific prior knowledge. We propose a general learning framework for modeling agent behavior in any multiagent system using only a handful of interaction data. Our framework casts agent modeling as a representation learning problem. Consequently, we construct a novel objective inspired by imitation learning and agent identification and design an algorithm for unsupervised learning of representations of agent policies. We demonstrate empirically the utility of the proposed framework in (i) a challenging high-dimensional competitive environment for continuous control and (ii) a cooperative environment for communication, on supervised predictive tasks, unsupervised clustering, and policy optimization using deep reinforcement learning.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

The Hanabi Challenge: A New Frontier for AI Research (2019, methodology): “Several MARL algorithms have been recently proposed that learn from models of others whether by using a training regime based on cognitive hierarchies [87], by defining a learning rule that shapes the anticipated learning of other agents [88], or by training architectures that incorporate agent iden”
Deep Reinforcement Learning for Swarm Systems (2018, background): “Finally, Grover et al. [2018] show a framework to model agent behavior as a representation learning problem.”
A survey and critique of multiagent deep reinforcement learning (2018, background): “Our conclusions of this work are that while the number of works in DRL and MDRL are notable and represent important milestones for AI, at the same time we acknowledge there are also open questions in both (deep) single-agent learning [81,151,211,328] and multiagent learning [116,172,195,242,245,360]”
Influencing Long-Term Behavior in Multiagent Reinforcement Learning (2022, unknown): “Other related works are the opponent modeling frameworks that infer opponents’ policies and condition an agent’s policy on the inferred information about others (He et al., 2016; Raileanu et al., 2018; Grover et al., 2018; Wen et al., 2019).”
Local Information Agent Modelling in Partially-Observable Environments (2021, methodology): “Grover et al. [2018] proposed an encoder-decoder method for modelling the agent’s policy.”
A Policy Gradient Algorithm for Learning to Learn in Multiagent Reinforcement Learning (2020, background): “The literature on opponent modeling, for instance, infers opponents’ behaviors and conditions an agent’s policy on the inferred behaviors of others (He et al., 2016; Raileanu et al., 2018; Grover et al., 2018).”
Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms (2019, background): “Other important goals include how to learn without over-fitting certain agents (He et al., 2016; Lowe et al., 2017; Grover et al., 2018), and how to learn robustly with either malicious/adversarial or failed/dysfunctional learning agents (Gao et al.”
ToM2C: Target-oriented Multi-agent Communication and Cooperation with Theory of Mind (2021, methodology): “Motivated by this, machine learning researchers have spent efforts on developing the machine ToM (Rabinowitz et al., 2018) or modeling opponents (He et al., 2016) for multiagent learning (Raileanu et al., 2018; Grover et al., 2018; Lim et al., 2020).”
Towards Open Ad Hoc Teamwork Using Graph-based Policy Learning (2020, methodology): “Our usage of LSTMs for type inference aligns with previous approaches for creating fixed-length embeddings of agents (Grover et al., 2018; Rabinowitz et al., 2018).”
Dealing with Non-Stationarity in Multi-Agent Deep Reinforcement Learning (2019, methodology): “…representations and initialization values from meta-learning (Al-Shedivat et al. , 2018) and approaches which learn opponent representations (Grover et al. , 2018; Tacchetti et al. , 2019; Ra-binowitz et al. , 2018) can be viewed as transferred knowledge which might result in quicker…”
Distributed Asynchronous Optimization with Unbounded Delays: How Slow Can You Go? (2018, background): “…(Recht et al., 2011; Yun et al., 2014; Petroni & Querzoni, 2014), to training deep neural networks (Dean et al., 2012a; Zhang et al., 2013; Paine et al., 2013; Zhang et al., 2015; Grover et al., 2015; Jiang et al., 2017) and representation learning (Grover & Leskovec, 2016; Grover et al., 2018a;b).”
Markov Decision Processes (2004, unknown): “, 2020), Opponent Modeling (Grover et al., 2018), Fast Adaptation (Raileanu et al.”
Multi-agent deep reinforcement learning: a survey (2021, background): “2018b) with imitation learning (Grover et al. 2018; Le et al. 2017), in social dilemmas (Jaques et al.”