State Representation Learning for Control: An Overview

By Timothée Lesort, Natalia Díaz Rodríguez, Jean-François Goudou, David Filliat. 2018

381 citations recorded by the source corpus.

Representation learning algorithms are designed to learn abstract features that characterize data. State representation learning (SRL) focuses on a particular kind of representation learning where learned features are in low dimension, evolve through time, and are influenced by actions of an agent. The representation is learned to capture the variation in the environment generated by the agent's actions; this kind of representation is particularly suitable for robotics and control scenarios. In particular, the low dimension characteristic of the representation helps to overcome the curse of dimensionality, provides easier interpretation and utilization by humans and can help improve performance and speed in policy learning algorithms such as reinforcement learning. This survey aims at covering the state-of-the-art on state representation learning in the most recent years. It reviews different SRL methods that involve interaction with the environment, their implementations and their applications in robotics control tasks (simulated or real). In particular, it highlights how generic learning objectives are differently exploited in the reviewed algorithms. Finally, it discusses evaluation methods to assess the representation learned and summarizes current and future lines of research.

Subfields: Artificial Intelligence, Reinforcement Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

In Representation & Self-Supervised Learning

Citations in the corpus

DisCoRL: Continual Reinforcement Learning via Policy Distillation (2019, methodology): “Another method we also exploit is to ﬁrst learn a state representation [Lesort et al., 2018] to compress the observation into a low dimensional embedding and secondly learn the policy on top of this representation.”
On the Transfer of Inductive Bias from Simulation to the Real World: a New Disentanglement Dataset (2019, background): “sequence modeling, curiosity-based exploration or even domain adaptation in reinforcement learning [45, 25, 37, 16, 18, 29, 47].”
Deep Reinforcement Learning for Autonomous Driving: A Survey (2020, background): “A good learned state representation should be Markovian; i.e. it should encode all necessary information to be able to select an action based on the current state only, and not any previous states or actions [60], [62].”
Continual Reinforcement Learning in 3D Non-stationary Environments (2019, methodology): “In this way, while still using a single end-to-end model and constructing more robust features [15, 22], it would be possible to integrate the benefit of both approaches when learning continuously.”
Continual Learning for Robotics (2019, background): “The ﬁrst challenge for RL is the extraction of representations to understand and compact what is relevant from the input data [82].”
Defining Benchmarks for Continual Few-Shot Learning (2020, methodology): “More recently, generative models such as GANs and VAEs (Lesort et al., 2018; 2019b) have been proposed to represent previously seen data as weights of a neural network.”
SOLAR: Deep Structured Latent Representations for Model-Based Reinforcement Learning (2018, methodology): “Utilizing representation learning within model-based RL has been studied in a number of previous works [23], including using embeddings for state aggregation [36], dimensionality reduction [31], self-organizing maps [37], value prediction [32], and deep auto-encoders [22, 11, 43, 15].”
Deep Dynamical Modeling and Control of Unsteady Fluid Flows (2018, background): “Our work most closely parallels work in state representation learning (SRL) [40], which focuses on learning low-dimensional features that are useful for modeling the time evolution of high-dimensional systems.”
DREAM Architecture: a Developmental Approach to Open-Ended Learning in Robotics (2020, background): “This issue is addressed in a variety of ways, such as learning a lower dimensionality representation [111, 53], or using learned models of the environment [112].”
S-TRIGGER: Continual State Representation Learning via Self-Triggered Generative Replay (2019, background): “…using RL. Previous work [14] has shown this can be more efﬁcient than using directly sensory states for RL. Recent advances in representation learning [2] have brought successful tools for learning a model of the surroundings of the agent, a ﬁeld known as State Representation Learning (SRL) [12].”
Continual State Representation Learning for Reinforcement Learning using Generative Replay (2018, neutral): “We propose to use a class of generative models often used for SRL, Variational Auto-Encoders (VAEs), combined to a Continual Learning (CL) approach to overcome this issue.”
SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning (2018, positive): “However, one significant advantage of representation and model learning over alternative approaches, such as modelfree RL, is the potential for transferring knowledge across multiple tasks where the underlying system dynamics do not change (Lesort et al., 2018).”
Explainability in Deep Reinforcement Learning (2020, methodology): “A general notion is that the knowledge gained from the auxiliary task objective must be useful for downstream tasks.”
Overview and comparative study of dimensionality reduction techniques for high dimensional data (2020, neutral): “Moreover, 2D or 3D will provide better interpretation and visualization of data [27] .”
An Overview of Natural Language State Representation for Reinforcement Learning (2020, methodology): “Lesort et al. (2018), for instance, presented a review on State Representation Learning algorithms and evaluation methods for control scenarios.”