Hidden-Mode Markov Decision Processes for Nonstationary Sequential Decision Making

By Samuel Ping-Man Choi, Dit-Yan Yeung, Nevin Lianwen Zhang. 2001

73 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

Citations in the corpus

Towards Continual Reinforcement Learning: A Review and Perspectives (2020, methodology): “HM-MDPs (Choi, Yeung, and Zhang, 2000) and DP-MDPs (Xie et al., 2021) are special CRL problems where α ⊆ {p, r} and β ∈ {stationary, passive}.”
Online Fast Adaptation and Knowledge Accumulation: a New Approach to Continual Learning (2020, background): “Similarly, it can represent unobserved variables in partially observable Markov decision processes (POMDPs) [32], hidden contexts in hidden-mode Markov decision processes [12], or tasks’ variations in robotics [75, 27, 82].”
Lifelong Hyper-Policy Optimization with Multiple Importance Sampling Regularization (2021, background): “…case, many works focus on the detection of abrupt changes in the dynamics (da Silva et al. 2006; Hadoux, Beynier, and Weng 2014) or on scenarios where the non-stationarity arises from switching between stationary dynamics, where the number of such dynamics is known (Choi, Yeung, and Zhang 2000).”
A Kernel-Based Approach to Non-Stationary Reinforcement Learning in Metric Spaces (2020, background): “For instance, Choi et al. (2000) propose a model where the MDP varies according to a sequence of tasks whose changes form a Markov chain.”
Sequential Decision-Making under Non-stationary Environments via Sequential Change-point Detection (2014, background): “Recent works propose to formalize non-stationary environments as a set of contexts (also called modes or modules) where a context corresponds to a possible stationary dynamics of the environments [4, 6, 14].”
A Survey of Learning in Multiagent Environments: Dealing with Non-Stationarity (2017, background): “Different MDPs along with its transition probabilities form an HM-MDP which can be seen as a special case of a POMDP (Choi et al., 2001).”
Deep Reinforcement Learning amidst Lifelong Non-Stationarity (2020, background): “While relaxing this assumption leads, in the worst case, to a POMDP, there is potentially additional structure that may be exploited under the HM-MDP (Choi et al., 2000) assumption of infrequent, discrete, unobserved shifts in the task parameters.”
Opportunities for multiagent systems and multiagent reinforcement learning in traffic control (2009, methodology): “Partial models have been used for the purpose of dealing with non-stationarity in [19,25].”
Online Fast Adaptation and Knowledge Accumulation (OSAKA): a New Approach to Continual Learning (2020, background): “Other frameworks study some of the features in silos but when methods are tested in less realistic settings some methods perform better than they should [12].”
Deep Reinforcement Learning amidst Continual Structured Non-Stationarity (2021, background): “A strongly related setting is the hidden-mode MDP (Choi et al., 2000), which augments the MDP with a latent parameter that evolves via a hidden Markov model with a discrete number of states.”
Prediction-Based Multi-Agent Reinforcement Learning in Inherently Non-Stationary Environments (2017, methodology): “Behaviour Change Detection Detected through change in rewards [Choi et al. 2001; Doya et al. 2002] RL-CD: Uses memory for known models, learns new models online [Silva et al.”
Sequoia: A Software Framework to Unify Continual Learning Research (2021, methodology): “Some cases where this setting has been studied are [12, 52] for RL and [7, 23, 22] for SL.”