The Optimal Control of Partially Observable Markov Processes over a Finite Horizon

By R. Smallwood, E. Sondik. 1973

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Kernel Mean Embedding of Distributions: A Review and Beyonds (2016, methodology): “Traditional approaches for solving the Bellman equation such as dynamic programming (Bellman 2003), parametric methods (Engel et al. 2003), Monte Carlo methods (Sutton and Barto 1998), and piecewise linear and convex (PWLC)-based methods (Smallwood and Sondik 1973) have several drawbacks.”
Informing sequential clinical decision-making through reinforcement learning: an empirical study (2010, background): “This problem is not without precedent in the reinforcement learning literature, where the Partially Observable Markov Decision Process (POMDP) paradigm was developed to handle cases in which the observation does not fully reveal the state of the system (Smallwood and Sondik 1973; Monahan 1982; Kaelb”
Infinite-Horizon Policy-Gradient Estimation (2001, methodology): “Fortunately, the observation history may be summarized in the form of a belief state(the current distribution over states), which is itself updated based only upon the current observation, and knowledge of which is sufficient for optimal behaviour (Smallwood & Sondik, 197 3; Sondik, 1978). An extens”
Joint Design and Separation Principle for Opportunistic Spectrum Access in the Presence of Sensing Errors (2007, background): “It is shown in [3] that the value function of an unconstrained POMDP with finite action space is piece-wise linear and can be solved via linear programming.”
A Bayesian Approach for Learning and Planning in Partially Observable Markov Decision Processes (2011, background): “A key result by Smallwood and Sondik (1973) shows that the optimal value function for a finite-horizon POMDP is piecewise-linear and convex.”
Convex Optimization: Algorithms and Complexity (2015, unknown): “The Partially Observable Markov Decision Process (POMDP) is an extension of MDP to the case where the state of the system is not necessarily observable [Astrom, 1965, Smallwood and Sondik, 1973, Kaelbling et al., 1998].”
Decision-Theoretic Planning: Structural Assumptions and Computational Leverage (1999, methodology): “As mentioned above, a POMDP can be viewed as a FOMDP with an infinite-size state space consisting of probability distributions over , each distribution representing the agent’s state of belief at a point in time [2, 125].”
Planning, Learning and Coordination in Multiagent Decision Processes (1996, background): “2 Given an MDP, an agent ought to adopt an optimal policy that maximizes the expected rewards accumulated as it 1Partially observable processes are much more realistic in many cases [13], but are much less tractable computationally [51].”
Risk Sensitive Markov Decision Processes (1997, result): “The following generalize similar structural results for the standard riskneutral case [1], [4], [13], [29], [32].”
Computing Optimal Policies for Partially Observable Decision Processes Using Compact Representations (1996, methodology): “However, Sondik's key observation that k-stage-to-go value functions are p.l.c., and thus finitely representable, also provides a means to finitely represent policies (albeit indirectly).”
A survey of algorithmic methods for partially observed Markov decision processes (1991, methodology): “Some numerical examples drawn from Sondik's dissertation and Smallwood and Sondik [ 29 ] demonstrate the technique.”
Active Policy Learning for Robot Planning and Exploration under Uncertainty (2007, unknown): “This calls for an active learning approach.”
Dynamic Programming for Partially Observable Stochastic Games (2004, methodology): “Exact DP algorithms for POMDPs rely on Smallwood and Sondik’s [23] proof that the DP operator preserves the piecewise linearity and convexity of the value function.”
Optimal Value of Information in Graphical Models (2009, background): “The problem of optimizing decision theoretic value of information can be naturally formalized as a (finite-horizon) Partially Observable Markov Decision Process (POMDP, Smallwood & Sondik, 1973 ).”
SARSOP: Efficient Point-Based POMDP Planning by Approximating Optimally Reachable Belief Spaces (2008, background): “POMDPs are a principled approach for planning and decision making under uncertainty [6, 17], but they are notoriously hard to solve [7, 9].”
Markovian Models for Sequential Data (2004, methodology): “In the control and reinforcement learning literature, similar models have been called Partially Observable Markov Decision Processes [104, 105, 4].”
Active Policy Learning for Robot Planning and Exploration under Uncertainty (2008, background): “That is, the discretized partially observed Markov decision process is too large for stochastic dynamic programming [23].”