A survey of algorithmic methods for partially observed Markov decision processes

By W. Lovejoy. 1991

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

A Survey of Embodied AI: From Simulators to Research Tasks (2021, positive): “In this section, the non-baseline approaches in visual exploration are typically formalized as partially observed Markov decision processes (POMDPs) [61].”
Reinforcement Learning: A Survey (1996, methodology): “POMDP Approach Another strategy consists of using hidden Markov model (HMM) techniques to learn a model of the environment, including the hidden state, then to use that model to construct a perfect memory controller (Cassandra, Kaelbling, & Littman, 1994; Lovejoy, 1991; Monahan, 1982).”
Neural Predictive Belief Representations (2018, methodology): “We consider Partially Observable Markov Decision Processes (POMDPs; Lovejoy, 1991; Cassandra, 1998) as a general framework to deal with partially-observable and stochastic environments with actions.”
Anytime Point-Based Approximations for Large POMDPs (2006, methodology): “When we do this, we can show that the pointbased solution is always be a lower-bound on the exact solution (Lovejoy, 1991a).”
Hierarchical control and learning for markov decision processes (1998, background): “General algorithms for this type of partially observable MDP (POMDP)(Lovejoy, 1991) are complicated and can be very ineecient.”
Hidden-Mode Markov Decision Processes for Nonstationary Sequential Decision Making (2001, methodology): “…brie y discuss how the latter can be achieved by a variant ofthe Baum-Welch algorithm, and then develop a similar algorithm for HM-MDP.3.1 Learning a POMDP ModelTraditional research in POMDP (Monahan 1982; Lovejoy 1991) assumes a knownenvironment model and is concerned with nding an optimal policy.”
Acting Optimally in Partially Observable Stochastic Domains (1994, methodology): “…2 B Let t = 0 Loop t := t + 1 For all b 2 B V t b = max a2A b; a + P b 0 2B b; a; b 0 V t,1 b 0 Until jV t b , V t,1 bj for all b 2 B This algorithm is guaranteed to converge in a nite number of iterations and results in a policy that is within 2=1, of the optimal policy Bellman 1957; Lovejoy 1991.”
Particle methods for change detection, system identification, and control (2004, methodology): “For finite state-space POMDPs, one can solve finite and infinite horizon problems using value iteration [41].”
Reinforcement learning for factored Markov decision processes (2002, methodology): “Chapter 3 details a method for learning compact representations of the core process of a POMDP while simultaneously approximating the belief-state action-value function.”
Planning treatment of ischemic heart disease with partially observable Markov decision processes (2000, methodology): “Then, the Bellman’s equation 4 holds and either exact [29, 16] or approximate dynamic programming techniques [25, 19, 10] developed for the standard POMDPs can be applied.”
Stochastic dynamic programming with factored representations (2000, methodology): “In Section 5 we present an empirical analysis of these methods and suggest the types of problems for which it is likely to work well, unlikely to work well, and what possible approaches may help with the latter.”
World Discovery Models (2019, unknown): “Although we do not explicitly use the corresponding terminology, this process can be formalised in terms of Partially Observable Markov Decision Processes (POMDPs; Lovejoy, 1991; Cassandra, 1998).”
Robotic mapping: a survey (2003, methodology): “In the field of robot mapping, the single dominating scheme for integrating such temporal data is known as Bayes filter [45], which is highly related to Kalman filters [49, 64], hidden Markov models [78], dynamic Bayes networks [82], and partially observable Markov decision processes [48, 59, 66, 93”
A Framework for Sequential Planning in Multi-Agent Settings (2005, methodology): “rk Our work draws from prior research on partially observable Markov decision processes, which recently gained a lot of attention within the AI community (Smallwood & Sondik, 1973; Monahan, 1982; Lovejoy, 1991; Hausktecht, 1997; Kaelbling et al., 1998; Boutilier et al., 1999; Hauskrecht, 2000). ”
Time Limits in Reinforcement Learning (2017, methodology): “In effect, a time-unaware agent has to act in a partially observable Markov decision process (POMDP) (Lovejoy, 1991) where states that only differ by their remaining time appear identical.”