Intrinsically motivated neuroevolution for vision-based reinforcement learning

By Giuseppe Cuccu, Matthew D. Luciw, J. Urgen Schmidhuber, Faustino J. Gomez. 2011

26 citations recorded by the source corpus.

Neuroevolution, the artificial evolution of neural networks, has shown great promise on continuous reinforcement learning tasks that require memory. However, it is not yet directly applicable to realistic embedded agents using high-dimensional (e.g. raw video images) inputs, requiring very large networks. In this paper, neuroevolution is combined with an unsupervised sensory pre-processor or compressor that is trained on images generated from the environment by the population of evolving recurrent neural network controllers. The compressor not only reduces the input cardinality of the controllers, but also biases the search toward novel controllers by rewarding those controllers that discover images that it reconstructs poorly. The method is successfully demonstrated on a vision-based version of the well-known mountain car benchmark, where controllers receive only single high-dimensional visual images of the environment, from a third-person perspective, instead of the standard two-dimensional state vector which includes information about velocity.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

From Pixels to Torques: Policy Learning with Deep Dynamical Models (2015, background): “to control a slot car based on raw images, employing deep architectures for ﬁnding compact low-dimensional representations. Other examples of deep learning in the context of RL on image data include (Cuccu et al., 2011; Koutnik et al., 2013). These approaches have in common that they try to estimate”
ViZDoom: A Doom-based AI research platform for visual reinforcement learning (2016, background): “Other works in this area include teaching mobile robots with visual-based Qlearning [10], learning policies with deep auto-encoders and batch-mode algorithms [19], neuroevolution for a vision-based version of the mountain car problem [6], and compressed neuroevolution with recurrent neural networks ”
A Brief Survey of Deep Reinforcement Learning (2017, positive): “eschewed the commonly used backpropagation algorithm in favour of evolutionary algorithms [32, 19, 53], which are gradient-free policy search algorithms.”
Learning States Representations in POMDP (2013, background): “For example, (Schmidhuber, 1990) and (Cuccu et al., 2011) present models using recurrent neural networks (RNN) to learn a controller for a given task.”
Auto-Agent-Distiller: Towards Efficient Deep Reinforcement Learning Agents via Neural Architecture Search (2020, methodology): “Value-based methods [78, 57] aim to build a value function, which subsequently allows the deﬁnition of a policy; and policy-based methods [67] directly model the policy to maximize the expected return in either a gradient-free [23, 13] or gradient-based way [67].”
Maximizing Fun by Creating Data with Easily Reducible Subjective Complexity (2013, background): “More recently implemented variants deal with applications to vision-based reinforcement learning/evolutionary search (Cuccu et al. 2011; Luciw et al. 2011), active learning of currently easily learnable functions (Ngo et al. 2011), blackbox optimization (Schaul et al. 2011b), and detection of "…”
Autoencoder-augmented neuroevolution for visual doom playing (2017, methodology): “The idea of compressing visual input so that evolutionary algorithms may be used has been demonstrated before in fairly simple two-dimensional environments [3].”