Predictive Representations of State

By Michael L. Littman, Richard S. Sutton, Satinder Singh. 2001

615 citations recorded by the source corpus.

Subfields: Reinforcement Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Representation & Self-Supervised Learning

Citations in the corpus

The Predictron: End-To-End Learning and Planning (2016, background): “The notion of learning about many predictions of the future relates to work on predictive state representations (PSRs; Littman et al., 2001), general value functions (GVFs; Sut-ton et al., 2011), and nexting (Modayil et al., 2012).”
Recurrent Environment Simulators (2017, background): “…forms part of deliberative planning systems used by the brain (Niv, 2009); and in reinforcement learning, the ability to imagine the future evolution of an environment is needed to form predictive state representations (Littman et al., 2002) and for Monte Carlo planning (Sutton & Barto, 1998).”
Hilbert Space Embeddings of Predictive State Representations (2013, background): “Predictive State Representations (PSRs) [3] are generalizations of IO-HMMs that have attracted interest because they can have greater representational capacity for a fixed model dimension.”
Learning to Act by Predicting the Future (2016, background): “In another recent report, Blundell et al. (2016) considered a nonparametric approach to control and conducted experiments in a three-dimensional labyrinth.”
DARLA: Improving Zero-Shot Transfer in Reinforcement Learning (2017, background): “Past attempts to build RL agents with strong domain adaptation performance highlighted the importance of learning good internal representations of raw observations (Finn et al., 2015; Raffin et al., 2017; Pan & Yang, 2009; Barreto et al., 2016; Littman et al., 2001).”
Successor Features for Transfer in Reinforcement Learning (2016, neutral): “’s [13] predictive state representation (PSR).”
APRIL: Active Preference-learning based Reinforcement Learning (2012, neutral): “However, the literature underlines that complex policies can hardly be expressed in the state × action space for tractability reasons [21].”
Contextual Decision Processes with low Bellman rank are PAC-Learnable (2016, methodology): “Unlike rank-based notions in PSRs (Littman et al., 2001) and multiplicity automata (Schützenberger, 1961), Bellman rank depends both on the process and the class F .”
Closing the learning-planning loop with predictive state representations (2009, background): “Predictive State Representations (PSRs) [11] and the closely related Observable Operator Models (OOMs) [8] are generalizations of POMDPs that have attracted interest because they both have greater representational capacity than POMDPs and yield representations that are at least as compact [21, 4].”
State Representation Learning in Robotics: Using Prior Knowledge about Physical Interaction (2014, background): “[17], define states as success probabilities for a set of tests, where a test is a prediction about future observations conditioned on future actions.”
Tensor decompositions for learning latent variable models (2012, methodology): “Notably, another class of methods, based on subspace identification (Overschee and Moor, 1996) and observable operator models/multiplicity automata (Schützenberger, 1961; Jaeger, 2000; Littman et al., 2001), have been proposed for a number of latent variable models.”
Neural Predictive Belief Representations (2018, background): “Predictive State Representations (PSRs; Littman & Sutton, 2002) are one such expressive and compact representation, in terms of tests on the POMDP (Rivest & Schapire, 1993).”
A Geometric Perspective on Optimal Representations for Reinforcement Learning (2019, background): “…notion of augmenting an agent with side predictions regarding the world is not new, with roots in TD models (Sutton, 1995), predictive state representations (Littman et al., 2002) and the Horde architecture (Sutton et al., 2011), which itself is based on Selfridge (1959)’s Pandemonium architecture.”
Reinforcement Learning of POMDPs using Spectral Methods (2016, background): “Perkins (2002) proposes a Monte-Carlo approach to action-value estimation and it shows convergence to locally optimal memoryless policies.”
The Barbados 2018 List of Open Issues in Continual Learning (2018, background): “Auxiliary questions can be constructed bottom-up by identifying other signals, and predicting (or controlling) accumulations of those signals [6, 12].”
Probabilistic Recurrent State-Space Models (2018, background): “Representations range from SSMs (Van Overschee & De Moor, 2012) over Predictive State Representations (PSRs) (Littman & Sutton, 2002; Singh et al., 2003; Rudary & Singh, 2004) to autoregressive models (Murray-Smith & Girard, 2001; Gi-rard et al., 2003; Likar & Kocijan, 2007; Billings, 2013), as…”