Reinforcement Learning on Slow Features of High-Dimensional Input Streams

By Robert A. Legenstein, Niko Wilbert, Laurenz Wiskott. 2010

95 citations recorded by the source corpus.

Humans and animals are able to learn complex behaviors based on a massive stream of sensory information from different modalities. Early animal studies have identified learning mechanisms that are based on reward and punishment such that animals tend to avoid actions that lead to punishment whereas rewarded actions are reinforced. However, most algorithms for reward-based learning are only applicable if the dimensionality of the state-space is sufficiently small or its structure is sufficiently simple. Therefore, the question arises how the problem of learning on high-dimensional data is solved in the brain. In this article, we propose a biologically plausible generic two-stage learning system that can directly be applied to raw high-dimensional input streams. The system is composed of a hierarchical slow feature analysis (SFA) network for preprocessing and a simple neural network on top that is trained based on rewards. We demonstrate by computer simulations that this generic architecture is able to learn quite demanding reinforcement learning tasks on high-dimensional visual input streams in a time that is comparable to the time needed when an explicit highly informative low-dimensional state-space representation is given instead of the high-dimensional visual input. The learning speed of the proposed architecture in a task similar to the Morris water maze task is comparable to that found in experimental studies with rats. This study thus supports the hypothesis that slowness learning is one important unsupervised learning principle utilized in the brain to form efficient state representations for behavioral learning.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Molecular Biology, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Molecular Biology

In Reinforcement Learning

Citations in the corpus

Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images (2015, background): “Enforcing temporal slowness during learning has previously been found to be a good proxy for learning representations in reinforcement learning [39, 40], and representation learning from videos [41].”
Deep learning in neural networks: An overview (2014, background): “The compact codes (instead of the high-dimensional raw data) can be fed into an RL machine, whose job thus may become much easier (Legenstein et al., 2010; Cuccu et al., 2011), just like SL may profit from UL, e.”
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (2015, methodology): “2) [42] used as sensory preprocessors to create less redundant sensory codes [118, 138, 142, 46].”
Online Evolution of Deep Convolutional Network for Vision-Based Reinforcement Learning (2014, background): “), is to combine action learning with an unsupervised learning (UL) preprocessor or “compressor” which provides a lower-dimensional feature vector that the agent receives as input instead of the raw observation [5, 8, 12, 17, 19–21].”
Unsupervised Learning of Visual Representations using Videos (2015, methodology): “, 2014) and as feature extractor for reinforcement learning (Legenstein et al., 2010).”
Learning state representation for deep actor-critic control (2016, methodology): “In [20], [18] this assumption is used to learn a mapping between visual observations and a state representation that gradually changes over time.”
Incremental Slow Feature Analysis: Adaptive Low-Complexity Slow Feature Updating from High-Dimensional Input Streams (2012, background): “, 2010), as a preprocessor for reinforcement learning (Legenstein et al., 2010; Kompella et al., 2011b), and learning of place-cells, head-direction cells, grid-cells, and spatial view cells from high-dimensional visual input (Franzius et al.”
Learning state representations with robotic priors (2015, methodology): “This method has been used to identify a representation of body postures of a humanoid robot [8] as well as for solving reinforcement learning tasks with visual observations [21].”
Construction of approximation spaces for reinforcement learning (2013, methodology): “Hierarchical application of quadratic SFA has been proposed to solve this problem (Wiskott and Sejnowski, 2002; Legenstein et al., 2010).”
Autonomous Learning of State Representations for Control: An Emerging Field Aims to Autonomously Learn State Representations for Reinforcement Learning Agents from Their Real-World Sensor Observations (2015, methodology): “This is why only few learning algorithms have been successful in this setting [5, 16, 25, 26].”
Evolving deep unsupervised convolutional networks for vision-based reinforcement learning (2014, background): “…of single-agent RL (i.e. TD [23], policy gradients [22], etc.), is to combine action learning with an unsupervised learning (UL) preprocessor or “compressor” which provides a lower-dimensional feature vector that the agent receives as input instead of the raw observation [5, 8, 11, 15, 17, 18, 19].”
Intrinsically motivated neuroevolution for vision-based reinforcement learning (2011, background): “TD, policy gradients, etc), where the agent simultaneously learns both the mapping from the compressed features to actions, and the features themselves from the observations provoked by the actions [1,4,11,13,14,16].”