Hilbert Space Embeddings of POMDPs

By Yu Nishiyama, Abdeslam Boularias, Arthur Gretton, Kenji Fukumizu. 2012

51 citations recorded by the source corpus.

A nonparametric approach for policy learning for POMDPs is proposed. The approach represents distributions over the states, observations, and actions as embeddings in feature spaces, which are reproducing kernel Hilbert spaces. Distributions over states given the observations are obtained by applying the kernel Bayes' rule to these distribution embeddings. Policies and value functions are defined on the feature space over states, which leads to a feature space expression for the Bellman equation. Value iteration may then be used to estimate the optimal value function and associated policy. Experimental results confirm that the correct policy is learned using the feature space representation.

Subfields: Artificial Intelligence, Civil and Structural Engineering, Computer Networks and Communications, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Civil and Structural Engineering

In Computer Networks and Communications

In Probabilistic & Bayesian Methods

Citations in the corpus

Kernel Mean Embedding of Distributions: A Review and Beyonds (2016, methodology): “In a state-space model, for example, we typically obtain a kernel mean estimate of the predictive distribution from the algorithm (Song et al. 2009, Nishiyama et al. 2012, McCalman et al. 2013).”
Characteristic Kernels and Infinitely Divisible Distributions (2014, background): “Keywords: Characteristic Kernel, Kernel Mean, Infinitely Divisible Distribution, Stable Distribution, GH Distribution”
Filtering with State-Observation Examples via Kernel Monte Carlo Filter (2013, background): “ diverse applications including hypothesis testing (Gretton et al., 2012, 2008), Bayesian inference (Song et al., 2009; Fukumizu et al., 2011), and reinforcement learning (Gru¨newa¨lder et al., 2012; Nishiyama et al., 2012). The RKHS framework has the following desirable properties: 1) it is complet”
Hilbert Space Embeddings of Predictive State Representations (2013, methodology): “This work was extended to POMDPs by Nishiyama et al. [13].”
The Nonparametric Kernel Bayes Smoother (2016, methodology): “Examples include the nonparametric kernel Bayes’ filter (nKB-filter) on discrete-time time-invariant state space models [10] and the Semiparametric kernel Monte-Carlo filter (KMC-filter) [16], as well as methods for dynamical system modeling and reinforcement learning applications [14, 21, 23, 4, 3,”
Kernel Embeddings of Conditional Distributions: A Unified Kernel Framework for Nonparametric Inference in Graphical Models (2013, methodology): “In this paper, we will review the recent development of nonparametric inference with kernel embeddings of conditional distributions [4, 5, 6, 7, 8, 9, 10, 11, 12], and provide a novel, unified kernel framework for addressing a broad class of challenging problems in graphical models.”
Kernel Bayes' rule: Bayesian inference with positive definite kernels (2013, methodology): “An alternative thresholding approach is proposed by Nishiyama et al. (2012), although its consistency remains to be established.”
Recovering Distributions from Gaussian RKHS Embeddings (2014, background): “, 2013), and reinforcement learning (Grünewälder et al., 2012; Nishiyama et al., 2012).”
Kernel Mean Estimation and Stein's Effect (2013, unknown): “In some applications of kernel mean embedding, one may want to interpret the weight β as a probability vector (Nishiyama et al., 2012).”
Non-parametric Policy Search with Limited Information Loss (2017, methodology): “Instead, similar to previous value function methods (Grünewälder et al., 2012b; Boots et al., 2013; Nishiyama et al., 2012), we will employ empirical conditional RKHS embeddings to obtain non-linear models for step-based reinforcement learning.”
Learning of Non-Parametric Control Policies with High-Dimensional State Features (2015, methodology): “In [Grünewälder et al., 2012b, Nishiyama et al., 2012] it is assumed that the state-action space can be sampled uniformly, which is unrealistic for real systems.”
Monte Carlo Filtering Using Kernel Embedding of Distributions (2014, methodology): “For example, we can naturally extend the kernel POMDP algorithm (Nishiyama et al. 2012) to the one with the sampling procedure.”
Smooth Operators (2013, methodology): “Besides covariate shift and Bayesian inference as discussed above, additional applications include spectral methods for inference in hidden Markov models, and reinforcement learning (Song et al., 2010; Grünewälder et al., 2012b; Nishiyama et al., 2012).”