Approximations of Dynamic Programs, I

By Ward Whitt. 1978

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Information-Theoretic Considerations in Batch Reinforcement Learning (2019, methodology): “Throughout the paper, we make novel connections to two subareas of RL: state abstractions (Whitt, 1978; Li et al., 2006) and PAC exploration under function approximation (Krishnamurthy et al.”
Near-Optimal Representation Learning for Hierarchical Reinforcement Learning (2018, background): “Bounding sub-optimality in abstracted MDPs has a long history, from early work in theoretical analysis on approximations to dynamic programming models (Whitt, 1978; Bertsekas & Castanon, 1989).”
An algebraic approach to abstraction in reinforcement learning (2004, background): “For a more complete description and the proofs see (Whitt, 1978).”
Feature-based methods for large scale dynamic programming (1995, methodology): “Whitt (1978) and Reetz (1977) analyzed approaches of reducing state space sizes, which lead to compact representations .”
Provably efficient RL with Rich Observations via Latent State Decoding (2019, background): “The most popular notion of abstraction in model-based RL is bisimulation (Whitt, 1978; Givan et al., 2003), which is more general than our setup since our context is sampled i.i.d. conditioned on the hidden state (the irrelevant factor discarded by a bisimulation may not be i.i.d.).”
Empirical Dynamic Programming (2013, background): “These ideas evolved independently in different fields, including the work of Werbos [48], Kushner and Clark [26] in control theory, Minsky [29], Barto, et al [4] and others in computer science, and Whitt in operations research [49, 50].”
Value Preserving State-Action Abstractions (2020, background): “The study of abstraction in RL has a long and exciting history, dating back to early work on approximating dynamic programs by Fox (1973) and Whitt (1978, 1979), along with the early work on hierar- 1In this work, we are uninterested in degenerate options (Bacon et al., 2017; Harb et al., 2018)…”
Towards Safe Policy Improvement for Non-Stationary MDPs (2020, background): “[63] W.”
Bounding Performance Loss in Approximate MDP Homomorphisms (2008, background): “Ravindran & Barto (2003) provided, based on a result from Whitt (1978), a bound on the difference in values between the optimal policy in the aggregated MDP and the lifted policy in the original MDP.”
Policy Error Bounds for Model-Based Reinforcement Learning with Factored Linear Models (2016, background): “…MBRL is “hard” because the derived policy cannot be good before the model approximates “reality” uniformly everywhere, we point out that already the ﬁrst ever bound derived for policy error in MBRL (due to Whitt (1978)) shows that the model has to be accurate only in an extremely localized way.”
Reinforcement Learning: An Introduction (1998, methodology): “) There is now a fairly extensive literature on function approximation methods and DP, such as multigrid methods and methods using splines and orthogonal polynomials (e.g., Bellman and Dreyfus, 1959; Bellman, Kalaba, and Kotkin, 1963; Daniel, 1976; Whitt, 1978; Reetz, 1977; Schweitzer and Seidmann, ”