Hilbert Space Embeddings of Predictive State Representations

By Byron Boots, Geoffrey J. Gordon, Arthur Gretton. 2013

97 citations recorded by the source corpus.

Predictive State Representations (PSRs) are an expressive class of models for controlled stochastic processes. PSRs represent state as a set of predictions of future observable events. Because PSRs are defined entirely in terms of observable data, statistically consistent estimates of PSR parameters can be learned efficiently by manipulating moments of observed training data. Most learning algorithms for PSRs have assumed that actions and observations are finite with low cardinality. In this paper, we generalize PSRs to infinite sets of observations and actions, using the recent concept of Hilbert space embed-dings of distributions. The essence is to represent the state as one or more nonparametric conditional embedding operators in a Reproducing Kernel Hilbert Space (RKHS) and leverage recent work in kernel methods to estimate, predict, and update the representation. We show that these Hilbert space embeddings of PSRs are able to gracefully handle continuous actions and observations, and that our learned models outperform competing system identification algorithms on several prediction benchmarks.

Subfields: Representation & Self-Supervised Learning, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

In Statistical Learning Theory

Citations in the corpus

Kernel Mean Embedding of Distributions: A Review and Beyonds (2016, methodology): “, there is a linear relationship between conditional probabilities of tests—it can still represent nonlinear dynamics (Boots et al. 2013).”
Characteristic Kernels and Infinitely Divisible Distributions (2014, background): “Keywords: Characteristic Kernel, Kernel Mean, Infinitely Divisible Distribution, Stable Distribution, GH Distribution”
A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms (2019, background): “Predictive state representations (PSRs) allow the robot to model the distributions over future observations based on the history of past observations [249, 349, 36, 357].”
Spectral Methods for Indian Buffet Process Inference (2014, background): “In particular, the work of [6, 5, 11, 21, 29] demonstrates that it is possible to infer latent variable structure accurately, despite the problem being nonconvex, thus exhibiting many local minima.”
The Nonparametric Kernel Bayes Smoother (2016, methodology): “Examples include the nonparametric kernel Bayes’ filter (nKB-filter) on discrete-time time-invariant state space models [10] and the Semiparametric kernel Monte-Carlo filter (KMC-filter) [16], as well as methods for dynamical system modeling and reinforcement learning applications [14, 21, 23, 4, 3,”
Non-parametric Policy Search with Limited Information Loss (2017, methodology): “Instead, similar to previous value function methods (Grünewälder et al., 2012b; Boots et al., 2013; Nishiyama et al., 2012), we will employ empirical conditional RKHS embeddings to obtain non-linear models for step-based reinforcement learning.”
Learning predictive models of a depth camera & manipulator from raw execution traces (2014, methodology): “This problem is far more difficult than the simulated problems explored in previous PSR work [10], [12]–[14].”
Initialization matters: Orthogonal Predictive State Recurrent Neural Networks (2018, methodology): “We use a ridge-regression parameter of 10(−2) (this is consistent with the values suggested in Boots et al. (2013); Downey et al. (2017)).”
Methods of Moments for Learning Stochastic Languages: Unified Presentation and Empirical Comparison (2014, methodology): “We did not examine the effect of large or continuous alphabets, as in those cases the performance of moment-methods are contingent upon the feature representation or kernel embedding employed (Song et al., 2010; Boots et al., 2013).”
Recovering Distributions from Gaussian RKHS Embeddings (2014, background): “, 2011), predictive state representations (Boots et al., 2013), and reinforcement learning (Grünewälder et al.”
Predictive State Recurrent Neural Networks (2017, methodology): “In two-stage regression we use a ridge parameter of 10(−2)n where n is the number of training examples (this is consistent with the values suggested in [8]).”
Learning Causal State Representations of Partially Observable Environments (2019, background): “In [4] the authors learn PSRs in Reproducing Kernel Hilbert Space extending the approach to continuous, potentially inﬁnite, action-observation processes.”
Kernel Bayes' rule: Bayesian inference with positive definite kernels (2013, methodology): “…in other papers; Nishiyama et al. (2012) propose a KBR approach to reinforcement learning in partially observable Markov decision processes, Boots et al. (2013) apply KBR to reinforcement learning with predictive state representations, and Nakagome et al. (2013) consider a KBR-based method…”