Proto-value Functions: A Laplacian Framework for Learning Representation and Control in Markov Decision Processes

By Sridhar Mahadevan, Mauro Maggioni. 2007

388 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Reinforcement Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

In Representation & Self-Supervised Learning

Citations in the corpus

DeepMDP: Learning Continuous Latent Space Models for Representation Learning (2019, background): “Other representation learning approaches take the perspective that an optimal representation contains features that allow for the linear parametrization of the optimal value function (Comanici & Precup, 2011; Mahadevan & Maggioni, 2007).”
Successor Features for Transfer in Reinforcement Learning (2016, background): “The resulting framework is an elegant extension of DP’s basic setting that provides a solid foundation for transfer in RL.”
Data-Efficient Hierarchical Reinforcement Learning (2018, background): “Goal-conditioned value functions [26, 38, 34, 1, 31] are actively explored outside the context of HRL.”
Learning Plannable Representations with Causal InfoGAN (2018, methodology): “For continuous observations, SC requires a distance function to build a connectivity graph, and previous studies [26, 40, 25] relied on the Euclidean distance, using either nearest neighbor to connect nodes, or exponentiated-distance weighted edges.”
State Representation Learning in Robotics: Using Prior Knowledge about Physical Interaction (2014, background): “Luciw and Schmidhuber [18] showed that slow feature analysis can approximate proto-value functions [19], which form a compact basis for all value functions.”
Transfer Learning for User Adaptation in Spoken Dialogue Systems (2016, background): “Representation transfer [37] is an active research area: one can find studies on option based transfers [20, 30], which are dependant on hierarchical reinforcement learning [36], on Proto-value functions [29, 15], or on representation transfer for Gaussian Processes Reinforcement Learning [16, 17, 5”
Regularization in reinforcement learning (2011, methodology): “Another similar basis generation method is a graph Laplacian-based approach [Mahadevan and Maggioni, 2007].”
Value Function Approximation in Reinforcement Learning Using the Fourier Basis (2011, background): “Recent work (Mahadevan and Maggioni, 2007) has focused on learning basis function sets from experience, removing the need to design an approximator but introducing the need to gather data to create one.”
A Geometric Perspective on Optimal Representations for Reinforcement Learning (2019, methodology): “We compare the value-based and AVF-based representations from the previous section ( VALUE and AVF ), and also proto-value functions (Mahadevan and Maggioni, 2007, PVF ).”
Autonomous Learning of State Representations for Control: An Emerging Field Aims to Autonomously Learn State Representations for Reinforcement Learning Agents from Their Real-World Sensor Observations (2015, methodology): “Proto-value functions (PVF, [34]) use Laplacian eigenmaps (LEM, [1]), a technique from spectral clustering [47], to provide a state representation X .”
Manifold Embeddings for Model-Based Reinforcement Learning under Partial Observability (2009, background): “Approaches for efficient function approximation, basis function construction, and discovery of embeddings has been the topic of significant investigations [3, 11, 20, 15, 13].”
Spectral Inference Networks: Unifying Deep and Spectral Learning (2018, methodology): “…state representations has been proposed as a method for learning a coordinate system of environments for planning and control (Boots et al., 2011), and when the transition function is symmetric its eigenfunctions are also known as proto-value functions (PVFs) (Mahadevan & Maggioni, 2007).”
Basis Function Discovery Using Spectral Clustering and Bisimulation Metrics (2011, methodology): “They are usually reflective only of the MDP transition model, rather than the entire MDP model (Mahadevan 2005).”
Transfer Learning for Reinforcement Learning Domains: A Survey (2009, background): “…state • a: disabling theNeutral action • p: the agent could describe the state by using extra state variables, such as the velocity on the previous timestep, but the agent only directly measures its current position and velocity • #: the agent may need to control two cars simultaneously on the hill”
Analyzing feature generation for value-function approximation (2007, methodology): “The .rst is based upon graph structures built up from observed trajectories through the state space (Mahadevan & Maggioni, 2006).”