Contextual Markov Decision Processes

By Assaf Hallak, Dotan Di Castro, Shie Mannor. arXiv.org, 2015

316 citations recorded by the source corpus.

We consider a planning problem where the dynamics and rewards of the environment depend on a hidden static parameter referred to as the context. The objective is to learn a strategy that maximizes the accumulated reward across all contexts. The new model, called Contextual Markov Decision Process (CMDP), can model a customer's behavior when interacting with a website (the learner). The customer's behavior depends on gender, age, location, device, etc. Based on that behavior, the website objective is to determine customer characteristics, and to optimize the interaction between them. Our work focuses on one basic scenario--finite horizon with a small known number of possible contexts. We suggest a family of algorithms with provable guarantees that learn the underlying models and the latent contexts, and optimize the CMDPs. Bounds are obtained for specific naive implementations, and extensions of the framework are discussed, laying the ground for future research.

Subfields: Artificial Intelligence, Management Science and Operations Research

View Paper: https://www.semanticscholar.org/paper/04955600df47c66a055591d927c024e6e9c72c61

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

Citations in the corpus

Block Contextual MDPs for Continual Learning (2021, background): “…assumptions: (i) the block assumption (Du et al., 2019) that addresses rich observations with irrelevant features and (ii) the contextual MDP (Hallak et al., 2015) assumption - MDPs with different dynamics and rewards share a common structure and a context that can describe the variation…”
Encouraging Physical Activity in Patients With Diabetes: Intervention Using a Reinforcement Learning System (2016, background): “Such an approach would require a larger population and would probably call for a different type of algorithm that takes into account contextual parameters as well leading to much better performance [12].”
CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agents (2021, methodology): “We refer the reader to (Zhang et al., 2018; Ghosh et al., 2021; Kirk et al., 2021), on which we base our discussion of generalization in RL using a Contextual MDP (CDMP) (Hallak et al., 2015).”
Continual Model-Based Reinforcement Learning with Hypernetworks (2020, unknown): “In this setting, a lot of works focus on quickly adapting to the change in dynamics to minimize online regret, as opposed to retaining performance on previously experienced dynamics [23], [24], [25].”
Multi-Task Reinforcement Learning with Context-based Representations (2021, background): “Contextual MDPs have been previously deﬁned and analyzed as a setting where side information is exploited for transfer across tasks (Hallak et al., 2015; Modi et al., 2017).”
VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning (2019, background): “Related to BAMDPs are contextual MDPs, where the task description is referred to as a context, on which the environment dynamics and rewards depend (Hallak et al., 2015; Jiang et al., 2017; Dann et al., 2018; Modi & Tewari, 2019).”
Contextual Decision Processes with low Bellman rank are PAC-Learnable (2016, background): “Hallak et al. [2015] have proposed a setting called Contextual MDPs, where a context refers to some static information that can be used to generalize across many similar MDPs.”
Offline Meta Reinforcement Learning (2020, unknown): “Related to this approach are contextual-MDPs [20, 23].”
RecSim: A Configurable Simulation Platform for Recommender Systems (2019, background): “Another set of sufﬁcient statistics that is commonly used in POMDP reinforcement learning is that of ﬁnite histories of observables.”