Discounted Dynamic Programming

By D. Blackwell. 1965

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Lawrence Carin
2. Yu Zhang
3. Xuejun Liao
4. J. Traub
5. John Rust
6. Henryk Wozniakowski
7. J. Adda
8. Russell Cooper

Citations in the corpus

• Deep Exploration via Randomized Value Functions (2017, neutral): “A discount γ approximates an effective planning horizon H ≃ 1 1−γ , but affords a solution to the discounted Bellman equation Qγ(s, a) = EM [R(s, a) + γmaxa′ Q ∗ γ(s ′, a′)] (Blackwell, 1965).”
• A survey of algorithmic methods for partially observed Markov decision processes (1991, background): “Blackwell [ 5 ]) policies can be restricted to stationary Markov policies 8: /7( X)---, A. For any V and W in B(/7(X)), define o(V, W)= sup{ I V(rr)- W(rr) l: ~r~/7(X)}.”
• On discounted dynamic programming with unbounded returns (2011, background): “The theory of stochastic dynamic programming (or Markov decision processes) with uncountable state space started with the fundamental work of Blackwell (1965).”
• Multi-task Reinforcement Learning in Partially Observable Stochastic Environments (2009, background): “These properties not only establish a more formal connection between the MV algorithm here and the traditional ML algorithm based on EM, they also shed light on the close relations between Theorem 6 and the policy improvement theorem of POMDP (Blackwell 1965).”
• Is There a Curse of Dimensionality for Contraction Fixed Points in the Worst Case (2002, background): “We take f x =max*x1 xm+ and obtain V s =max[ 1 s + E1V s m s + EmV s ] The associated fixed point equation, V = V , is known as Bellman’s equation, the fundamental equation underlying infinite horizon Markovian decision problems (see, e.g., Blackwell (1965) and Denardo (1967)).”
• Computable Markov-Perfect Industry Dynamics: Existence, Purification, and Multiplicity (2007, unknown): “Application of Blackwell’s sufficient conditions (theorem 5 of Blackwell (1965); see also theorem 3.3 on p. 54 of Stokey & Lucas (1989)) shows that H∗ n,u−n is a contraction with modulus β: First, inspection of equation (10) shows that Vn(ω) ≥ V̂n(ω) for all ω implies (H∗ n,u−nVn)(ω) ≥ (H∗ n,u−n V̂n”
• Is Leasing Greener Than Selling? (2012, methodology): “Because the per-period net utility is bounded and the strategy space is finite, the above problem can be solved by deriving the Bellman equation for consumer ˆ using backward induction (Blackwell 1965).”
• Dynamic Economics: Quantitative Methods and Applications (2003, background): “The fixed point argument proceeds by showing the T (W ) is a contraction using a pair of sufficient conditions from Blackwell (1965). These conditions are: (i) monotonicity and (ii) discounting of the mapping T (V ).”
• Modeling of risk-based inspection, maintenance and life-cycle cost with partially observable Markov decision processes (2005, background): “…and Infrastructure Engineering ISSN 1573-2479 print/ISSN 1744-8980 online # 2005 Taylor & Francis Ltd http://www.tandf.co.uk/journals DOI: 10.1080/15732470412331289305 of expected total rewards and the criterion of expected average rewards, respectively (Blackwell 1965, Derman 1966, 1970).”
• Optimal Replacement of GMC Bus Engines: An Empirical Model of Harold Zurcher (1987, background): “Blackwell (1968)) shows that under quite general conditions the solution to the class of infinite horizon Markovian decision problems takes the general form”
• Bandit processes and dynamic allocation indices (1979, background): “As shown by Blackwell (1965), the maximum total expected reward for any Markov decision process satisfies a dynamic programming functional equation.”