Relational Deep Reinforcement Learning

By Vinícius Flores Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David P. Reichert, Timothy P. Lillicrap, Edward Lockhart and 6 more. 2018

245 citations recorded by the source corpus.

We introduce an approach for deep reinforcement learning (RL) that improves upon the efficiency, generalization capacity, and interpretability of conventional approaches through structured perception and relational reasoning. It uses self-attention to iteratively reason about the relations between entities in a scene and to guide a model-free policy. Our results show that in a novel navigation and planning task called Box-World, our agent finds interpretable solutions that improve upon baselines in terms of sample complexity, ability to generalize to more complex scenes than experienced during training, and overall performance. In the StarCraft II Learning Environment, our agent achieves state-of-the-art performance on six mini-games -- surpassing human grandmaster performance on four. By considering architectural inductive biases, our work opens new directions for overcoming important, but stubborn, challenges in deep RL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning, Graph Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

In Graph Learning

Citations in the corpus

Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, background): “…(i.e., posthoc approximations), including tree-based explanations (Coppens et al., 2019; Liu et al., 2018), reward decomposition (Juozapaitis et al., 2019), autonomous explanation (Fukuchi et al., 2017; Hayes and Shah, 2017), and attention-based methods (Zambaldi et al., 2018; Mott et al., 2019).”
Reconciling deep learning with symbolic artificial intelligence: representing objects and relations (2019, methodology): “A network architecture incorporating stacked or recurrent relation or self-attention modules could be thought of as carrying out a series of logic-like reasoning steps [42,51].”
Relational inductive biases, deep learning, and graph networks (2018, methodology): “…model-free (Wang et al., 2018b) and model-based (Hamrick et al., 2017; Pascanu et al., 2017; Sanchez-Gonzalez et al., 2018) continuous control, for model-free reinforcement learning (Hamrick et al., 2018; Zambaldi et al., 2018), and for more classical approaches to planning (Toyer et al., 2017).”
Spatially Invariant Unsupervised Object Detection with Convolutional Neural Networks (2019, methodology): “Related work from computer vision includes (Cao and Fei-Fei 2007; Kim and Torralba 2009; Lee and Grauman 2009; Russell et al. 2006; Karpathy, Miller, and Fei-Fei 2013), as well as work on co-segmentation (Zhu et al. 2016) and unsupervised object recognition (Tuytelaars et al. 2010).”
Emergent Tool Use From Multi-Agent Autocurricula (2019, methodology): “[73] Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, et al.”
Relational recurrent neural networks (2018, methodology): “We study a variant of BoxWorld, which is a pixel-based, highly combinatorial reinforcement learning environment that demands relational reasoning-based planning, initially developed in [46].”
Relational Forward Models for Multi-Agent Learning (2018, positive): “…entities and a rich relational structure, like physics simulation, multi-object scenes, visual question answering and motion-capture data (Battaglia et al., 2016; Raposo et al., 2017; Santoro et al., 2017; Gilmer et al., 2017; Watters et al., 2017; Kipf et al., 2018; Zambaldi et al., 2018).”
A Survey of Deep Reinforcement Learning in Video Games (2019, methodology): “[98] introduce the relational DRL to StarCraft, which iteratively reasons about the relations between entities with self-attention, and uses it to guide a model-free RL policy.”
Deep Reinforcement Learning for Cyber Security (2019, background): “[30], game of Go [31], [32], real-time strategy game StarCraft II [33]–[36], 3-D multiplayer game Quake III Arena Capture the Flag [37], and teamwork game Dota 2 [38] to real-world applications, such as robotics [39], autonomous vehicles (AVs) [40], autonomous surgery [41], [42], natural language pr”
Neural Logic Reinforcement Learning (2019, methodology): “However, as a graph-based rela-Neural tional model was used (Zambaldi et al., 2018), the learned policy is not fully explainable and the rules expression is limited, different from the interpretable logic-represented policies learned in ours using DILP.”
Evolutionary Population Curriculum for Scaling Multi-Agent Reinforcement Learning (2020, unknown): “, 2018), represent relations (Zambaldi et al., 2018; Malysheva et al., 2018; Yang et al., 2019) and extract communication channels (Jiang & Lu, 2018).”
MEMO: A Deep Network for Flexible Combination of Episodic Memories (2020, methodology): “Neural networks are then used to aggregate or learn functions over graph components to perform supervised, semi-supervised, representation and reinforcement learning tasks (Kipf & Welling, 2016; Gilmer et al., 2017; Hamilton et al., 2017; Zambaldi et al., 2018).”
Unsupervised Grounding of Plannable First-Order Logic Representation from Images (2019, unknown): “Deep Relational RL (Zambaldi et al. 2018) combines Relation Networks and attention-based message passing.”
Better Set Representations For Relational Reasoning (2020, methodology): “nging. Typical differentiable methods directly map the input to latent features using a feedforward neural network and partition the latent features into a set representation for downstream reasoning [1,11,19,26]. However, this class of existing methods has a fundamental ﬂaw that prevents them from ”
TStarBots: Defeating the Cheating Level Builtin AI in StarCraft II in the Full Game (2018, methodology): “Following this setting, even the state-of-the-art RL algorithm can only achieve success in playing toy mini-games that has much shorter horizon than the full game [16].”
A Case for Object Compositionality in Deep Generative Models of Images (2018, methodology): “Following Zambaldi et al. (2018) we consider one or more “attention blocks” to compute interactions among the object representations.”
Graph Convolutional Reinforcement Learning for Multi-Agent Cooperation (2018, background): “Neural networks can operate on structured representations of a set of entities, non-locally compute interactions on a set of feature vectors, and perform relational reasoning via iterated message passing (Zambaldi et al., 2018).”
Acquisition of chess knowledge in AlphaZero (2021, background): “Symbolic approaches have been proposed for representing objects and relations and better utilisation of either background or acquired knowledge in RL agents [51, 52, 53, 54].”