Hidden Parameter Markov Decision Processes: A Semiparametric Regression Approach for Discovering Latent Task Parametrizations

By Finale Doshi-Velez, George Dimitri Konidaris. 2013

142 citations recorded by the source corpus.

Control applications often feature tasks with similar, but not identical, dynamics. We introduce the Hidden Parameter Markov Decision Process (HiP-MDP), a framework that parametrizes a family of related dynamical systems with a low-dimensional set of latent factors, and introduce a semiparametric regression approach for learning its structure from data. We show that a learned HiP-MDP rapidly identifies the dynamics of new task instances in several settings, flexibly adapting to task variation.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “Current work also typically assumes a stationary task state distribution for meta-learning (Doshi-Velez and Konidaris, 2013; Wang et al., 2016; Duan et al., 2016; Zintgraf, Shiarlis, Kurin, Hofmann, and Whiteson, 2018; Rakelly, Zhou, Quillen, Finn, and Levine, 2019; Zintgraf, Shiarlis, Igl,…”
Meta Reinforcement Learning with Latent Variable Gaussian Processes (2018, methodology): “A similar framework to ours is found in [11], called hidden parameter Markov decision processes (HiP-MDP), which parametrizes a family of related dynamics through a low dimensional latent embedding.”
AdaRL: What, Where, and How to Adapt in Transfer Reinforcement Learning (2021, background): “, HiMDPs (Doshi-Velez and Konidaris, 2016)) and recent follow-ups (Zhang et al.”
Robust and Efficient Transfer Learning with Hidden Parameter Markov Decision Processes (2017, methodology): “The Hidden Parameter Markov Decision Process (HiP-MDP) (Doshi-Velez and Konidaris 2013) was introduced as a formalization of these domains with two primary features.”
Local Information Agent Modelling in Partially-Observable Environments (2021, methodology): “Works on meta-RL and transfer learning focus on learning representations over tasks and use them to solve new and previously unseen tasks [Doshi-Velez and Konidaris, 2016, Hausman et al., 2018, Zhang et al., 2018, Rakelly et al., 2019, Zintgraf et al., 2019].”
Fast Adaptation to New Environments via Policy-Dynamics Value Functions (2020, methodology): “Another common approach for transfer across dynamics is model-based RL, which uses Gaussian processes (GPs) or Bayesian neural networks (BNNs) to estimate the transition function (Doshi-Velez & Konidaris, 2013; Killian et al., 2017).”
Direct Policy Transfer via Hidden Parameter Markov Decision Processes (2018, methodology): “Model-based reinforcement learning & HiP-MDPs Hidden parameter MDPs, or HIP-MDPs (Doshi-Velez and Konidaris, 2016; Killian et al., 2017), describe a family of Markov Decision Processes (MDPs) via the tuple {S,A,W, T,R, γ, PW }, where S ⊆ R is the set of continuous states s, A is the set of discrete ”
Transfer Learning Across Patient Variations with Hidden Parameter Markov Decision Processes (2016, background): “The Hidden Parameter Markov Decision Process (HiP-MDP) [9] formalizes the transfer learning task in the following way: first it assumes that any task instance can be fully parameterized by a bounded number of latent parameters w.”
Deep Reinforcement Learning amidst Continual Structured Non-Stationarity (2021, methodology): “The Bayes-adaptive MDP formulation (BAMDP) (Duff, 2002; Ross et al., 2008), as well as the hidden parameter MDP (HiP-MDP) (Doshi-Velez & Konidaris, 2016) consider an MDP with unknown parameters governing the reward and dynamics, which we aim to infer online over the course of one episode.”
VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning (2019, result): “…of hidden parameter (HiP-) MDPs assumes that there is a set of low-dimensional latent factors which define a family of related dynamical systems (with shared reward structure), similar to the assumption we make in Equation (5) (Doshi-Velez & Konidaris, 2016; Killian et al., 2017; Yao et al., 2018).”
Block Contextual MDPs for Continual Learning (2021, methodology): “Zhang et al. (2021b) also uses task metrics to learn a context space, but focus on the rich observation version of the Hidden-Parameter MDP (HiP-MDP) setting (Doshi-Velez & Konidaris, 2013) with access to task ids.”
Deep Reinforcement Learning amidst Lifelong Non-Stationarity (2020, methodology): “Following the formulation of the Hidden Parameter MDP (HiP-MDP) [10], a set of unobserved task parameters z ∈ Z defines the dynamics ps(st+1|st,at; z) and reward function r(st,at; z) for each task.”
Generalized Hidden Parameter MDPs Transferable Model-based RL in a Handful of Trials (2020, methodology): “We extend Hidden Parameter MDPs introduced in (Doshi-Velez and Konidaris 2016) to the more general case including reward modeling and accounting for potentially multiple factors of variation.”