Schema Networks: Zero-shot Transfer with a Generative Causal Model of Intuitive Physics

By Ken Kansky, Tom Silver, David A. Mély, Mohamed Eldawy, Miguel L'azaro-Gredilla, Xinghua Lou, N. Dorfman, Szymon Sidor, Scott Phoenix, Dileep George. 2017

246 citations recorded by the source corpus.

The recent adaptation of deep neural network-based methods to reinforcement learning and planning domains has yielded remarkable progress on individual tasks. Nonetheless, progress on task-to-task transfer remains limited. In pursuit of efficient and robust generalization, we introduce the Schema Network, an object-oriented generative physics simulator capable of disentangling multiple causes of events and reasoning backward through causes to achieve goals. The richly structured architecture of the Schema Network can learn the dynamics of an environment directly from data. We compare Schema Networks with Asynchronous Advantage Actor-Critic and Progressive Networks on a suite of Breakout variations, reporting results on training efficiency and zero-shot generalization, consistently demonstrating faster, more robust learning and better transfer. We argue that generalizing from limited data and learning causal relationships are essential abilities on the path toward generally intelligent systems.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence (2020, background): “, (Kansky et al., 2017) showed this in a compelling way by tinkering with Breakout; even minor changes—such as moving the paddle up a few pixels up—led to drastic reductions in performance.”
Neural Logic Machines (2019, methodology): “Examples include OOMDP (Diuk et al., 2008; Kansky et al., 2017), symbolic models for learning in interactive domains (Pasula et al.”
On the Binding Problem in Artificial Neural Networks (2020, background): “(c) Neural network based agents trained on Breakout, fail to generalize to slight variations of the game such as a shifted paddle or an added middle wall (Kansky et al., 2017).”
Relational inductive biases, deep learning, and graph networks (2018, background): “, 2018), developing model-based approaches with an emphasis on abstraction (Kansky et al., 2017; Konidaris et al., 2018; Zhang et al., 2018; Hay et al., 2018), investing more heavily in meta-learning (Wang et al.”
Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions (2018, background): “…with the compositionality of symbolic programs in modeling physics (Battaglia et al., 2016; Chang et al., 2016), playing games (Denil et al., 2017; Kansky et al., 2017), learning algorithms (Bošnjak et al., 2017; Cai et al., 2017; Li et al., 2016; Reed & De Freitas, 2015), visual understanding…”
Relational Deep Reinforcement Learning (2018, background): “Unfortunately, deep RL models still face important limitations, namely, low sample efficiency and a propensity not to generalize to seemingly minor changes in the task [4, 5, 6, 7].”
Meta-learning autoencoders for few-shot prediction (2018, background): “This class of problems is important in many areas, for example, learning and predicting physics [8, 26, 25, 3], reinforcement learning of games [15, 10], unsupervised learning of videos [20], and applications such as self-driving, where we cannot enumerate and train with all environments that the…”
Continual Reinforcement Learning in 3D Non-stationary Environments (2019, background): “However, as shown in the past, reinforcement learning agents are quite fragile also with respect to minor environmental changes [18].”
Imagination-Augmented Agents for Deep Reinforcement Learning (2017, background): “[50] learn symbolic networks models of the environment and use them for planning, but are given the relevant abstractions from a hand-crafted vision system.”
Deep Learning: A Critical Appraisal (2018, background): “…one about the alleged victory of Google’s Jeff In the same paper, Vicarious proposed an alternative to deep learning called schema networks (Kansky et al., 2017) 7 that can handle a number of variations in the Atari game Breakout, albeit apparently without the multi-game generality of…”
A Brief Survey of Deep Reinforcement Learning (2017, neutral): “Recent work with (non-deep) generative causal models demonstrated superior generalisation over standard DRL algorithms [72, 96] in some benchmarks [8], achieved by reasoning about causes and effects in the environment [47].”
Combined Reinforcement Learning via Abstract Representations (2018, methodology): “As can be seen in Fig 3, using techniques such as tSNE (Maaten and Hinton 2008) are inefficient to represent a meaningful low-dimensional representation of this task7.”
A Perspective on Objects and Systematic Generalization in Model-Based RL (2019, methodology): “RL methods that leverage the combinatorics of objects and relations have shown similar beneﬁts in terms of systematic generalization (Zambaldi et al., 2019), sample efﬁ-ciency (Diuk et al., 2008), and transferring skills and knowledge across domains (Kansky et al., 2017).”
Cross-View Policy Learning for Street Navigation (2019, background): “[19] proposed Schema Networks to transfer experience from one scenario to other similar scenarios that exhibit repeatable structure and sub-structure.”
Reward Learning from Narrated Demonstrations (2018, background): “[28] showed how object-factorized state representations and dynamics can generalize across environmental variations, in contrast to frame-centric policies.”
Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense (2020, background): “This results in a significant challenge for transfer learning under today’s dominant machine learning paradigm [143,144].”
Transfer Learning for Related Reinforcement Learning Tasks via Image-to-Image Translation (2018, background): “One work transferring between modiﬁed versions of the same game using zero-shot transfer is the schema networks (Kansky et al., 2017).”