Eigenoption Discovery through the Deep Successor Representation

By Marlos C. Machado, Clemens Rosenbaum, Xiaoxiao Guo, Miao Liu, G. Tesauro, Murray Campbell. 2017

172 citations recorded by the source corpus.

Options in reinforcement learning allow agents to hierarchically decompose a task into subtasks, having the potential to speed up learning and planning. However, autonomously learning effective sets of options is still a major challenge in the field. In this paper we focus on the recently introduced idea of using representation learning methods to guide the option discovery process. Specifically, we look at eigenoptions, options obtained from representations that encode diffusive information flow in the environment. We extend the existing algorithms for eigenoption discovery to settings with stochastic transitions and in which handcrafted features are not available. We propose an algorithm that discovers eigenoptions while learning non-linear state representations from raw pixels. It exploits recent successes in the deep reinforcement learning literature and the equivalence between proto-value functions and the successor representation. We use traditional tabular domains to provide intuition about our approach and Atari 2600 games to demonstrate its potential.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Hybrid Reinforcement Learning with Expert State Sequences (2019, background): “Oh et al. (2015) and Machado et al. (2018) trained deep neural network models to predict the next frame or successor representation in Atari games with good effect.”
Spectral Inference Networks: Unifying Spectral Methods With Deep Learning (2018, methodology): “We used the same convolutional network architecture as [22], a batch size of 32 and RMSProp with a learning rate of 1e-4 for 300k iterations, and updated the target network every 10k iterations.”
A Geometric Perspective on Optimal Representations for Reinforcement Learning (2019, unknown): “Speciﬁcally, our PVF representation consists in the top k right-singular vectors of the successor representation ( I − γP π ) − 1 for π the uniformly random policy, as suggested by Machado et al. (2018); Be-hzadian and Petrik (2018, see also appendix, Figure 11).”
Exploration in Deep Reinforcement Learning: A Survey (2022, methodology): “This was later improved in [103] by replacing the handcrafted adjacency matrix with successor representations.”
The Laplacian in RL: Learning Representations with Efficient Approximations (2018, methodology): “C Derivation of (5) (Switching the notation u and v in the second term gives the same quantity as the ﬁrst term) D Additional results and experiment details et al., 2017a) or a representation learned by a forward prediction model (Machado et al., 2017b).”
Task-Agnostic Dynamics Priors for Deep Reinforcement Learning (2019, methodology): “We perform empirical evaluations on three different platforms - a suite of 2D games (PhysWorld), a 3D partially observable environment, and a stochastic version of Atari games (Machado et al., 2017a).”
Discovering Options for Exploration by Minimizing Cover Time (2019, methodology): “Deﬁnition 1 (option): An option o is deﬁned by a triple: ( I , π, β ) where: • I ⊆ S is a set of states where the option can initiate, • π : S → Pr( A ) is a policy, Many previous approaches propose methods to generate options based on heuristics and demonstrate the effectiveness in experimental eva”
Variational Option Discovery Algorithms (2018, background): “Machado et al. [2017a], Machado et al. [2017b], and Liu et al. [2017] learn eigenoptions , options derived from the graph Laplacian associated with the MDP.”
Decoupling Dynamics and Reward for Transfer Learning (2018, methodology): “We combine ideas from many areas of previous research, especially those of successor features (Dayan, 1993; Barreto et al., 2016a; Machado et al., 2017), which also uses a decoupling mechanism in the value function to transfer across reward functions (but not dynamics) (Barreto et al., 2016b;…”
Spectral Inference Networks: Unifying Deep and Spectral Learning (2018, methodology): “The use of PVFs for discovering subgoals in reinforcement learning has been investigated in (Machado et al., 2017) and combined with function approximation in (Machado et al., 2018), though using a less rigorous approach to eigenfunction approximation than SpIN.”
Learning Structures: Predictive Representations, Replay, and Generalization (2020, methodology): “This is why count-based representation learning, such as deep successor representation learning and successor feature learning, have been shown to support option discovery [28,76 ,77].”
Count-Based Exploration with the Successor Representation (2018, methodology): “Finally, recent work (Kulkarni et al., 2016, Machado et al., 2018b) have shown successor features can be learned jointly with the feature representation itself.”