Relational Forward Models for Multi-Agent Learning

By Andrea Tacchetti, H. Francis Song, Pedro A. M. Mediano, Vinícius Flores Zambaldi, János Kramár, Neil C. Rabinowitz, Thore Graepel, Matthew Botvinick, Peter W. Battaglia. 2018

85 citations recorded by the source corpus.

The behavioral dynamics of multi-agent systems have a rich and orderly structure, which can be leveraged to understand these systems, and to improve how artificial agents learn to operate in them. Here we introduce Relational Forward Models (RFM) for multi-agent learning, networks that can learn to make accurate predictions of agents' future behavior in multi-agent environments. Because these models operate on the discrete entities and relations present in the environment, they produce interpretable intermediate representations which offer insights into what drives agents' behavior, and what events mediate the intensity and valence of social interactions. Furthermore, we show that embedding RFM modules inside agents results in faster learning systems compared to non-augmented baselines. As more and more of the autonomous systems we develop and interact with become multi-agent in nature, developing richer analysis tools for characterizing how and why agents make decisions is increasingly necessary. Moreover, developing artificial agents that quickly and safely learn to coordinate with one another, and with humans in shared environments, is crucial.

Subfields: Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Learning to Simulate Complex Physics with Graph Networks (2020, background): “…et al., 2018; Mrowca et al., 2018; Li et al., 2019; Sanchez-Gonzalez et al., 2019), as well as non-physical systems, such as multi-agent dynamics (Tacchetti et al., 2018; Sun et al., 2019), algorithm execution (Velikovi et al., 2020), and other dynamic graph settings (Trivedi et al., 2019;…”
Recurrent Independent Mechanisms (2019, background): “…scenes, and motion-capture data, and multiagent systems (Scarselli et al., 2008; Bronstein et al., 2017; Watters et al., 2017; Raposo et al., 2017; Santoro et al., 2017; Gilmer et al., 2017; Van Steenkiste et al., 2018; Kipf et al., 2018; Battaglia et al., 2018; Tacchetti et al., 2018).”
Deep Coordination Graphs (2019, background): “, 2019) or as a joint critic in actor-critic frameworks (Tacchetti et al., 2019; Liu et al., 2019; Malysheva et al., 2019).”
Graph Convolutional Reinforcement Learning (2018, unknown): “Relational Forward Models (RFM) (Tacchetti et al., 2019) use supervised learning to predict the actions of all other agents based on global state.”
Know Your Action Set: Learning Action Relations for Reinforcement Learning (2022, background): “…with a rich relational structure, such as morphological control (Wang et al., 2018; Sanchez-Gonzalez et al., 2018; Pathak et al., 2019), multi-agent RL (Tacchetti et al., 2019), physical construction (Hamrick et al., 2018), and structured perception in games like StarCraft (Zambaldi et al., 2018).”
Dealing with Non-Stationarity in Multi-Agent Deep Reinforcement Learning (2019, methodology): “Architectures such as graph neural networks (Tacchetti et al. , 2019), feed-forward neural networks (Grover et al. , 2018), and recurrent neural networks (Rabinowitz et al. , 2018) can be used to produce the representations.”
Neural Production Systems (2021, background): “GNNs model pairwise interactions between all the entities hence they can be termed as capturing dense interactions (Scarselli et al., 2008; Bronstein et al., 2017; Watters et al., 2017; Van Steenkiste et al., 2018; Kipf et al., 2018; Battaglia et al., 2018; Tacchetti et al., 2018).”
Local Information Agent Modelling in Partially-Observable Environments (2021, unknown): “Tacchetti et al. [2019] proposed relational forward models to model agents using graph neural networks.”
Multi-agent deep reinforcement learning: a survey (2021, background): “Further works on constructing models can be found in cooperative tasks (Barde et al. 2019; Tacchetti et al. 2019; Zheng et al. 2018b) with imitation learning (Grover et al. 2018; Le et al. 2017), in social dilemmas (Jaques et al. 2019; Letcher et al. 2019), and by predicting behaviors from…”
Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning (2021, background): “Graph neural networks (GNN) [Gilmer et al., 2017, Tacchetti et al., 2018, Battaglia et al., 2018, Kipf et al., 2019] is a widely adopted relational model that have a factorized representation of variables and models pairwise interactions between objects while being permutation invariant.”
Towards Open Ad Hoc Teamwork Using Graph-based Policy Learning (2020, methodology): “To handle dynamic team sizes, the joint action value model and agent model are both based on graph neural network (GNN) architectures (Tacchetti et al., 2019; Böhmer et al., 2020) which have proven useful for dealing with changing input sizes (Hamilton et al.”