Decoupling feature extraction from policy learning: assessing benefits of state representation learning in goal based robotics

By Antonin Raffin, Ashley Hill, Kalifou René Traoré, Timothée Lesort, Natalia Díaz Rodríguez, David Filliat. 2018

62 citations recorded by the source corpus.

Scaling end-to-end reinforcement learning to control real robots from vision presents a series of challenges, in particular in terms of sample efficiency. Against end-to-end learning, state representation learning can help learn a compact, efficient and relevant representation of states that speeds up policy learning, reducing the number of samples needed, and that is easier to interpret. We evaluate several state representation learning methods on goal based robotics tasks and propose a new unsupervised model that stacks representations and combines strengths of several of these approaches. This method encodes all the relevant features, performs on par or better than end-to-end learning with better sample efficiency, and is robust to hyper-parameters change.

Subfields: Artificial Intelligence, Computational Theory and Mathematics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

Citations in the corpus

Deep Reinforcement Learning for Autonomous Driving: A Survey (2020, background): “it removes noise and decreases the domain’s size as shown in [61].”
Continual Learning for Robotics (2019, background): “In particular, this is due to the low data efficiency of RL algorithms (Raffin et al., 2019).”
Continual Learning in Deep Networks: an Analysis of the Last Layer (2021, background): “Decoupling the feature extractor from a sub-network has shown to be helpful, e.g., in reinforcement learning [20, 26, 31].”
Acquisition of chess knowledge in AlphaZero (2021, background): “Representation learning in RL agents [45, 46, 47, 48, 49, 50] can be beneficial when used to learn low-dimensional representations for states, policies and actions, hoping to meaningfully capture the variations in agent’s environment and disentangle the contributing factors.”
Lifelong Learning and Personalization in Long-Term Human-Robot Interaction (LEAP-HRI) (2021, background): “prediction [14], manipulation [15]–[19], dialogue [20], navigation [1], [16], [21]–[23] and learning socially appropriate robot behaviors [24], [25].”
Explainability in Deep Reinforcement Learning (2020, methodology): “This method consists of using a custom decomposed reward DQN by deﬁning a vector-valued reward function, where each component is the reward for a certain type so that actions can be compared in terms of trade-oﬀs among the types.”
S-TRIGGER: Continual State Representation Learning via Self-Triggered Generative Replay (2019, background): “Previous work (Raffin et al., 2019) has shown this can be more efficient than using directly sensory states for RL.”
DREAM Architecture: a Developmental Approach to Open-Ended Learning in Robotics (2020, background): “Arrows represent model learning and inference, dashed frames represent losses computation, rectangles are state representations, circles are real observed data, and squares are model predictions [64].”
Mid-Level Visual Representations Improve Generalization and Sample Efficiency for Learning Visuomotor Policies (2018, background): “…the data distribution being simpler than the raw pixels (unsupervised approaches like the autoen-coder family [34, 83, 41, 48] and Generative Adversarial Networks [77, 23, 57, 15]) or they exploit some known structure (so-called self-supervised approaches [91, 56, 55, 72, 84, 52, 28, 74, 13, 61]).”