Combined Reinforcement Learning via Abstract Representations

By Vincent François-Lavet, Yoshua Bengio, Doina Precup, Joelle Pineau. 2018

99 citations recorded by the source corpus.

In the quest for efficient and robust reinforcement learning methods, both model-free and model-based approaches offer advantages. In this paper we propose a new way of explicitly bridging both approaches via a shared low-dimensional learned encoding of the environment, meant to capture summarizing abstractions. We show that the modularity brought by this approach leads to good generalization while being computationally efficient, with planning happening in a smaller latent state space. In addition, this approach recovers a sufficient low-dimensional representation of the environment, which opens up new strategies for interpretable AI, exploration and transfer learning.

Subfields: Reinforcement Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Representation & Self-Supervised Learning

Citations in the corpus

DeepMDP: Learning Continuous Latent Space Models for Representation Learning (2019, methodology): “As (Francois-Lavet et al., 2018) notes, if only the latent transition loss was minimized, then the optimal solution is indeed φ : S Ñ 0 so that s P perfectly predicts φpPps, aqq.”
Learning First-Order Symbolic Representations for Planning from the Structure of the State Space (2019, background): “Deep reinforcement learning (DRL) methods [30], on the other hand, generate policies over high-dimensional perceptual spaces like images, without using any prior symbolic knowledge [19, 10, 15].”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “With rapid progress in deep RL, recent work (Zhang et al., 2018b; François-Lavet et al., 2019) has focused on building an abstract representation with a low-dimensional representation of relevant features for non-stationary settings.”
A Geometric Perspective on Optimal Representations for Reinforcement Learning (2019, methodology): “Ratitch and Precup (2004), Foster and Dayan (2002), Menache et al.”
Lifelong Hyper-Policy Optimization with Multiple Importance Sampling Regularization (2021, background): “2019) or by extracting meaningful information in the form of abstract concepts (Zhang, Satija, and Pineau 2018; Francois-Lavet et al. 2019).”
Contrastive Learning of Structured World Models (2019, background): “By establishing a connection between contrastive learning of state abstractions (François-Lavet et al., 2018; Thomas et al., 2018) and relational graph embeddings (Nickel et al.”
An Introduction to Deep Reinforcement Learning (2018, unknown): “Another approach to zero-shot transfer is to use algorithms that enforce states that relate to the same underlying task but have different renderings to be mapped into an abstract state that is close (Tzeng et al., 2015; François-Lavet et al., 2018).”
An Overview of Natural Language State Representation for Reinforcement Learning (2020, unknown): “Therefore, sys- 1See e.g. McCallum (1996); Kaelbling et al. (1998); Finney et al. (2002); Van Otterlo (2002); Morales (2004); Roy et al. (2005); Frommberger (2008); Mahmud (2010); Maillard et al. (2011); Ortiz et al. (2018); François-Lavet et al. (2019).”
Learning Action Representations for Reinforcement Learning (2019, methodology): “Auxiliary Tasks: Previous works showed empirically that supervised learning with the objective to predict a component of a transition tuple (s, a, r, s′) from the others, can be useful as an auxiliary method to learn state representations (Jaderberg et al., 2016; François-Lavet et al., 2018) or to o”
Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learning (2020, background): “Other work that strives to learn a latent agent state representation include (DeepMDP Gelada et al., 2019) and (CRAR François-Lavet et al., 2019) that try to learn a transition model entirely in latent space.”
The Value Equivalence Principle for Model-Based Reinforcement Learning (2020, background): “This is an instantiation of the problem known as representation learning [43, 21, 9, 18, 45, 44, 19, 7].”