Robust and Efficient Transfer Learning with Hidden Parameter Markov Decision Processes

By Taylor W. Killian, George Dimitri Konidaris, Finale Doshi-Velez. 2017

114 citations recorded by the source corpus.

We introduce a new formulation of the Hidden Parameter Markov Decision Process (HiP-MDP), a framework for modeling families of related tasks using low-dimensional latent embeddings. Our new framework correctly models the joint uncertainty in the latent parameters and the state space. We also replace the original Gaussian Process-based model with a Bayesian Neural Network, enabling more scalable inference. Thus, we expand the scope of the HiP-MDP to applications with higher dimensions and more complex dynamics.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Reinforcement Learning in Healthcare: A Survey (2019, methodology): “Other approaches applied various kinds of TRL techniques so as to take advantage of the prior information from previously learned transition models [146], [147] or learned policy [148].”
Meta Reinforcement Learning with Latent Variable Gaussian Processes (2018, methodology): “This work was extended in [18], replacing the GP basis functions with a Bayesian neural network.”
Transfer Learning in Deep Reinforcement Learning: A Survey (2020, unknown): “decision making [166], [167].”
Direct Policy Transfer via Hidden Parameter Markov Decision Processes (2018, methodology): “We do not compare to average model or online model-free learning with just the environment because Killian et al. (2017) demonstrated that the HiP-MDP dominated those approaches in their work.”
VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning (2019, result): “…of hidden parameter (HiP-) MDPs assumes that there is a set of low-dimensional latent factors which define a family of related dynamical systems (with shared reward structure), similar to the assumption we make in Equation (5) (Doshi-Velez & Konidaris, 2016; Killian et al., 2017; Yao et al., 2018).”
Model Selection in Bayesian Neural Networks via Horseshoe Priors (2017, background): “The details of the dynamics settings of both domains are described in (Killian et al., 2017).”
Model-based Reinforcement Learning for Semi-Markov Decision Processes with Neural ODEs (2020, background): “Establishing effective treatment strategies for HIV-infected patients based on markers from blood tests can be cast as an RL problem [Ernst et al., 2006, Parbhoo et al., 2017, Killian et al., 2017].”
Structured Variational Learning of Bayesian Neural Networks with Horseshoe Priors (2018, methodology): “…good predictive uncertainty estimates is crucial is in model-based re-inforcement learning scenarios (e.g. (Depeweg et al., 2017; Gal et al., 2016b; Killian et al., 2017)): here, it is essential not only to have an estimate of what state an agent may be in after taking a particular action, but…”
Generalized Hidden Parameter MDPs Transferable Model-based RL in a Handful of Trials (2020, background): “Recent prior work on latent variable models of MDPs focused on models of dynamics with small discrete action spaces (Doshi-Velez and Konidaris 2016; Killian et al. 2017; Yao et al. 2018).”
Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning (2021, background): “Causality-based methods are a different way of tackling generalization, but current solutions do not scale to high-dimensional observation spaces (e.g., Killian et al., 2017; Perez et al., 2020; Zhang et al., 2020).”
Fast Adaptation to New Environments via Policy-Dynamics Value Functions (2020, methodology): “Another common approach for transfer across dynamics is model-based RL, which uses Gaussian processes (GPs) or Bayesian neural networks (BNNs) to estimate the transition function (Doshi-Velez & Konidaris, 2013; Killian et al., 2017).”