Deep auto-encoder neural networks in reinforcement learning

By Sascha Lange, Martin A. Riedmiller. 2010

387 citations recorded by the source corpus.

This paper discusses the effectiveness of deep auto-encoder neural networks in visual reinforcement learning (RL) tasks. We propose a framework for combining the training of deep auto-encoders (for learning compact feature spaces) with recently-proposed batch-mode RL algorithms (for learning policies). An emphasis is put on the data-efficiency of this combination and on studying the properties of the feature spaces automatically constructed by the deep auto-encoders. These feature spaces are empirically shown to adequately resemble existing similarities and spatial relations between observations and allow to learn useful policies. We propose several methods for improving the topology of the feature spaces making use of task-dependent information. Finally, we present first results on successfully learning good control policies directly on synthesized and real images.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Playing Atari with Deep Reinforcement Learning (2013, methodology): “Since Q maps historyaction pairs to scalar estimates of their Q-value, the history and the action have been used as inputs to the neural network by some previous approaches [20, 12].”
Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images (2015, methodology): “In the context of representation learning for control (see B¨ohmer et al. [21] for a recent review), deep autoencoders (ignoring state transitions) similar to those used in our comparison have been applied previously, e.g. by Lange and Riedmiller [22].”
Deep learning in neural networks: An overview (2014, methodology): “For example, NFQ (Riedmiller, 2005) was applied to real-world control tasks (Lange and Riedmiller, 2010; Riedmiller et al., 2012) where purely visual inputs were compactly encoded by deep autoencoders (Sec.”
A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, positive): “, multilayer neural networks were used to train the auto-encoder model (Lange and JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 9 Riedmiller, 2010; Lange et al., 2012; Finn et al., 2015). Lange and Riedmiller (2010) deﬁned the deep-ﬁtted Q algorithm to reduce the dimension of the raw inp”
ViZDoom: A Doom-based AI research platform for visual reinforcement learning (2016, background): “Other works in this area include teaching mobile robots with visual-based Qlearning [10], learning policies with deep auto-encoders and batch-mode algorithms [19], neuroevolution for a vision-based version of the mountain car problem [6], and compressed neuroevolution with recurrent neural networks ”
Control of Memory, Active Perception, and Action in Minecraft (2016, methodology): “Neural networks have been used to learn features for RL tasks for a few decades (e.g., Tesauro, 1995 and Lange & Riedmiller, 2010).”
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (2015, methodology): “For example, NFQ [201] was applied to real-world control tasks [138, 202] where purely visual inputs were compactly encoded in hidden layers of deep autoencoders [245, Sec.”
Go-Explore: a New Approach for Hard-Exploration Problems (2019, background): “Additional options include unsupervised techniques such as networks that autoencode [53] or predict future states, and other auxiliary tasks such as pixel control [54].”
Learning to Play in a Day: Faster Deep Reinforcement Learning by Optimality Tightening (2016, methodology): “The model-free episodic control approach evaluates state-action pairs based on episodic memory using k-nearest neighbors with hashing functions (Blundell et al., 2016).”
SOLAR: Deep Structured Latent Representations for Model-Based Reinforcement Learning (2018, methodology): “Utilizing representation learning within model-based RL has been studied in a number of previous works [23], including using embeddings for state aggregation [36], dimensionality reduction [31], self-organizing maps [37], value prediction [32], and deep auto-encoders [22, 11, 43, 15].”
Construction of approximation spaces for reinforcement learning (2013, methodology): “For example, in the context of navigation, Lange and Riedmiller (2010) employed a deep auto-encoder (Hinton and Osindero, 2006), Legenstein et al. (2010) hierarchical nonlinear slow feature analysis (SFA, Wiskott and Sejnowski, 2002) and Luciw and Schmidhuber (2012) incremental SFA (Kompella et…”
Bisimulation Makes Analogies in Goal-Conditioned Reinforcement Learning (2022, methodology): “Self-supervised RL methods rely on autoencoders (Lange and Riedmiller, 2010; Lange et al., 2012; Yarats et al., 2021b), contrastive objectives (van den Oord et al.”
Sequential Constant Size Compressors for Reinforcement Learning (2011, methodology): “Such stacks of auto-encoders have already been used as preprocessing for RL [13].”
Offline Reinforcement Learning from Images with Latent Space Models (2020, background): “…policies either from pixel inputs end-to-end (Haarnoja et al., 2018; Gelada et al., 2019; Singh et al., 2019; Kostrikov et al., 2020; Laskin et al., 2020; Han et al., 2020) or on top of unsupervised visual representations Lange and Riedmiller (2010); Ghadirzadeh et al. (2017); Nair et al. (2018).”
Autoencoder-augmented neuroevolution for visual doom playing (2017, methodology): “Autoencoder compression of the environment has been shown to be successful when combined with Q-learning, as well, for successful results in simple Atari environments before the current popularity of Deep-Q learning[16], as well as on simple real-world applications such a slot car control [17].”
Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agents (2017, background): “A plethora of recent research suggests that is a viable approach (Lange & Riedmiller, 2010; Kingma & Welling, 2013; Bellemare et al., 2016).”
Intrinsically motivated neuroevolution for vision-based reinforcement learning (2011, background): “TD, policy gradients, etc), where the agent simultaneously learns both the mapping from the compressed features to actions, and the features themselves from the observations provoked by the actions [1,4,11,13,14,16].”