Towards Deep Symbolic Reinforcement Learning

By Marta Garnelo, Kai Arulkumaran, Murray Shanahan. 2016

236 citations recorded by the source corpus.

Deep reinforcement learning (DRL) brings the power of deep neural networks to bear on the generic task of trial-and-error learning, and its effectiveness has been convincingly demonstrated on tasks such as Atari video games and the game of Go. However, contemporary DRL systems inherit a number of shortcomings from the current generation of deep learning techniques. For example, they require very large datasets to work effectively, entailing that they are slow to learn even when such datasets are available. Moreover, they lack the ability to reason on an abstract level, which makes it difficult to implement high-level cognitive functions such as transfer learning, analogical reasoning, and hypothesis-based reasoning. Finally, their operation is largely opaque to humans, rendering them unsuitable for domains in which verifiability is important. In this paper, we propose an end-to-end reinforcement learning architecture comprising a neural back end and a symbolic front end with the potential to overcome each of these shortcomings. As proof-of-concept, we present a preliminary implementation of the architecture and apply it to several variants of a simple video game. We show that the resulting system -- though just a prototype -- learns effectively, and, by acquiring a set of symbolic rules that are easily comprehensible to humans, dramatically outperforms a conventional, fully neural DRL system on a stochastic variant of the game.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

A simple neural network module for relational reasoning (2017, background): “However, a number of these approaches, such as deep learning, often struggle in data-poor problems where the underlying structure is characterized by sparse but complex relations [3, 11].”
Neural Logic Machines (2019, methodology): “Examples include OO-MDP (Diuk et al., 2008; Kansky et al., 2017), symbolic models for learning in interactive domains (Pasula et al., 2007), structured task deﬁnition by object-oriented instructions (Denil et al., 2017), and structured policy learning (Garnelo et al., 2016).”
Compositional Attention Networks for Machine Reasoning (2018, background): “However, deep learning approaches often struggle to perform well on tasks with a compositional and structured nature (Garnelo et al., 2016; Lake et al., 2017).”
Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, positive): “Combining connectionist and symbolic paradigms seems a favourable way to address this challenge [169,299,312,352,353] .”
COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration (2019, methodology): “However, most existing approaches require hand-crafting object representations or their relations (Cobo et al., 2013; Diuk et al., 2008; Garnelo et al., 2016; Lázaro-Gredilla et al., 2019).”
Relational inductive biases, deep learning, and graph networks (2018, methodology): “In the spirit of numerous recent examples of principled hybrids of structure-based methods and deep learning (e.g., Reed and De Freitas, 2016; Garnelo et al., 2016; Ritchie et al., 2016; Wu et al., 2017; Denil et al., 2017; Hudson and Manning, 2018), we see great promise in synthesizing new…”
Spatially Invariant Unsupervised Object Detection with Convolutional Neural Networks (2019, methodology): “…learning methods that reason in terms of objects have obtained objects either using ad hoc, task/environment speciﬁc object detection methods (Garnelo, Arulkumaran, and Shanahan 2016; Diuk, Cohen, and Littman 2008), or, in settings where it is possible, using object representations provided…”
Towards a Definition of Disentangled Representations (2018, background): “Furthermore, their performance often lacks the same level of robustness and generalisability that is characteristic of biological intelligence [52, 31, 57].”
Relational Deep Reinforcement Learning (2018, background): “Unfortunately, deep RL models still face important limitations, namely, low sample efficiency and a propensity not to generalize to seemingly minor changes in the task [4, 5, 6, 7].”
Programmatically Interpretable Reinforcement Learning (2018, background): “Many recent efforts in deep learning aim to make deep networks more interpretable (Montavon et al., 2017; Lipton, 2016; Garnelo et al., 2016; Zahavy et al., 2016; Shanahan, 2005; Lake et al., 2016).”
Conditional Neural Processes (2018, background): “Deep neural networks have enjoyed remarkable success in recent years, but they require large datasets for effective training (Lake et al., 2017; Garnelo et al., 2016).”
Life-Long Disentangled Representation Learning with Cross-Domain Latent Homologies (2018, background): “Finding semantic homologies between visually distinctive domains can remove the need to learn from scratch on every new environment and hence help with data efficiency – another major drawback of current deep learning approaches [16, 29].”
DARLA: Improving Zero-Shot Transfer in Reinforcement Learning (2017, background): “These include data inefficiency, the reactive nature and general brittleness of learnt policies to changes in input data distribution, and lack of model interpretability (Garnelo et al., 2016; Lake et al., 2016).”
Schema Networks: Zero-shot Transfer with a Generative Causal Model of Intuitive Physics (2017, positive): “Recent work has demonstrated one possible method for unsupervised entity construction using autoencoders (Garnelo et al., 2016).”
A Brief Survey of Deep Reinforcement Learning (2017, neutral): “search [108] and planning [118], a deeper integration with other traditional AI approaches promises benefits such as better sample complexity, generalisation and interpretability [30].”
Measuring abstract reasoning in neural networks (2018, background): “Generalisation in neural networks has been subject of lots of recent debate, with some emphasising the successes (LeCun et al., 2015) and others the failures (Garnelo et al., 2016; Lake & Baroni, 2017; Marcus, 2018).”
Learning by Abstraction: The Neural State Machine (2019, background): “…potentially misleading statistical associations instead of learning true causal relations [1, 42], strongly depend on large amounts of data and supervision [25, 51], and sometimes behave in surprising and 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.”
Unsupervised Model Selection for Variational Disentangled Representation Learning (2019, background): “…of state of the art approaches are still missing many basic yet important properties, such as fairness, data efﬁcient learning, strong generalisation beyond the training data distribution, or the ability to transfer knowledge between tasks (Lake et al., 2016; Garnelo et al., 2016; Marcus, 2018).”
Emerging Technology and Business Model Innovation: The Case of Artificial Intelligence (2019, background): “Marta Garnelo developed an end-to-end reinforcement learning architecture comprising a neural back-end and a symbolic frond-end with the potential to address the challenges [34].”