On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models

By J. Urgen Schmidhuber. 2015

123 citations recorded by the source corpus.

This paper addresses the general problem of reinforcement learning (RL) in partially observable environments. In 2013, our large RL recurrent neural networks (RNNs) learned from scratch to drive simulated cars from high-dimensional video input. However, real brains are more powerful in many ways. In particular, they learn a predictive model of their initially unknown environment, and somehow use it for abstract (e.g., hierarchical) planning and reasoning. Guided by algorithmic information theory, we describe RNN-based AIs (RNNAIs) designed to do the same. Such an RNNAI can be trained on never-ending sequences of tasks, some of them provided by the user, others invented by the RNNAI itself in a curious, playful fashion, to improve its RNN-based world model. Unlike our previous model-building RNN-based RL machines dating back to 1990, the RNNAI learns to actively query its model for abstract reasoning and planning and decision making, essentially "learning to think." The basic ideas of this report can be applied to many other cases where one RNN-like system exploits the algorithmic information content of another. They are taken from a grant proposal submitted in Fall 2014, and also explain concepts such as "mirror neurons." Experimental results will be described in separate papers.

Subfields: Artificial Intelligence, Reinforcement Learning, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

In Model Architectures

Citations in the corpus

The Predictron: End-To-End Learning and Planning (2016, background): “Schmidhuber (2015) discusses learning abstract models, but maintains separate losses for the model and a controller, and suggests training the model unsupervised to compactly encode the entire history of observations, through predictive coding.”
On the Binding Problem in Artificial Neural Networks (2020, background): “The ability to interact with the environment may additionally enable an (embodied) agent to autonomously acquire multi-exemplar data for a particular relation (Schmidhuber, 2015; Haber et al., 2018).”
World Models (2018, methodology): “An iterative training procedure (Schmidhuber, 2015a) is as follows: 1.”
Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions (2018, background): “Other interactions between a controller C and a model of the world M (implemented by R-NEM) as posed in Schmidhuber (2015) constitute further research directions. the difference between the prediction and the noisy ground-truth as input.”
Selective Experience Replay for Lifelong Learning (2018, background): “The idea of retraining on old experiences has been briefly suggested for a similar purpose in training recurrent neural networks (Schmidhuber 2015).”
Imagination-Augmented Agents for Deep Reinforcement Learning (2017, background): “The general idea of learning to leverage an internal model in arbitrary ways was also discussed by Schmidhuber [53].”
Automatically Composing Representation Transformations as a Means for Generalization (2018, background): “Indeed, often it is easier to solve a problem Pa by making an analogy to a problem Pb that one already knows how to solve, rather than retraining to solve Pb from scratch [74, 76].”
A Roadmap Towards Machine Intelligence (2015, background): “ the papers we just mentioned, also appears in Solomono (2002), a work which has much earlier roots in research on program induction (Solomono, 1964, 1997; Schmidhuber, 2004). Within this tradition, Schmidhuber (2015) reviews a large literature and presents some general ideas on learning that might ”
One Big Net For Everything (2018, methodology): “1 Introduction I will ﬁrst quickly summarize a few relevant concepts discussed in much more detail in previous reports [76, 77].”
Some Considerations on Learning to Explore via Meta-Reinforcement Learning (2018, background): “eedy exploration [15], or perhaps a simple scheme like Boltzmann exploration [6]. As a result of these shortcomings, a variety of new approaches to exploration in deep RL have recently been suggested [34, 9, 31, 18, 4, 19, 17, 8, 23, 17, 8, 26, 25, 32, 33, 12, 24]. In spite of these numerous efforts”
Predictive Information Accelerates Learning in RL (2020, methodology): “This work falls in another group, where future prediction is used as an auxiliary or representation learning method for model-free RL agents [1, 10, 14, 22, 29, 30, 33, 34, 35, 37].”
Learning to Predict Without Looking Ahead: World Models Without Forward Prediction (2019, methodology): “[2], similar to our work, directly optimizes the dynamics model against loss by differentiating through a planning procedure, and Schmidhuber [50] proposes a similar idea of improving the internal model using an RNN, although the RNN world model is initially trained to perform forward prediction.”
Unity: A General Platform for Intelligent Agents (2018, background): “This is often the case for human task acquisition in the real world, and the ability to learn new tasks over time is seen as a key-component of continual learning (Ring, 1994), and ultimately systems capable of artiﬁcial general intelligence (Schmidhuber, 2015; Schmidhuber, 2018).”
Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective (2019, background): “Ha and Schmidhuber (2018) and Schmidhuber (2015) propose solutions to the cheat-the-world-model problem.”
Model-based Reinforcement Learning for Semi-Markov Decision Processes with Neural ODEs (2020, methodology): “One might use an iterative training procedure of Algorithm 2 [Ha and Schmidhuber, 2018, Schmidhuber, 2015] for more complex environments, which interleaves exploration and learning.”
Learning a Driving Simulator (2016, background): “Other vision only models focused on low dimensional [15] or videos with simple textures, such as those of Atari games [14] [16].”
A Growing Long-term Episodic & Semantic Memory (2016, methodology): “Following [28], our system is unsupervised, and its goal is to compress its experiences, which is an uninterrupted stream of fixed width vectors.”
Reinforcement Learning Upside Down: Don't Predict Rewards - Just Map Them to Actions (2019, result): “…do not have desired rewards as command inputs, and typically the training depends on an RNN-based predictive world model M (which predicts rewards, among other things) to compute gradients for the RNN controller C. RL, however, does not depend at all on good reward predictions (compare [53, Sec.”