Dynamic Programming

By Ryuichi Oka, Aizu Wakamatsu. 2018

3,604 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Mathematical Optimization, Algorithms & Complexity

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Mathematical Optimization

In Algorithms & Complexity

Citations in the corpus

Addressing Function Approximation Error in Actor-Critic Methods (2018, neutral): “In Q-learning, the value function can be learned using temporal difference learning (Sutton, 1988; Watkins, 1989), an update rule based on the Bellman equation (Bellman, 1957).”
PGQ: Combining policy gradient and Q-learning (2016, positive): “The Bellman operator T π (Bellman, 1957) for policy π is defined as T Q(s, a) = E s′,r,b (r(s, a) + γQ(s′, b)), where the expectation is over next state s′ ∼ P (·, s, a), the reward r(s, a), and the action b from policy πs′ .”
Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents (2017, positive): “According to Bellman’s optimality equation (Bellman, 1957), the optimal value of executing action a in state s is”
Kernel Mean Embedding of Distributions: A Review and Beyonds (2016, positive): “Traditional approaches for solving the Bellman equation such as dynamic programming (Bellman 2003), parametric methods (Engel et al. 2003), Monte Carlo methods (Sutton and Barto 1998), and piecewise linear and convex (PWLC)-based methods (Smallwood and Sondik 1973) have several drawbacks.”
Deep Reinforcement Learning for Autonomous Driving: A Survey (2020, neutral): “This problem is commonly referred to in the literature as the “curse of dimensionality”, a term originally coined by Bellman [37].”
Reinforcement Learning in Healthcare: A Survey (2019, neutral): “To involve the action information, the Q value is used to represent the optimal value of each state-action pair, Dynamic programming (DP) [18] is a collection of model-based techniques to compute an optimal policy given a complete description of an MDP model.”
Reinforcement Learning via Fenchel-Rockafellar Duality (2020, neutral): “This abstraction gives rise to the Bellman recurrence, which characterizes the optimal value function and behavior policy through a dynamic programming (DP) view of the RL problem (Bellman, 1966).”
Off-Policy Deep Reinforcement Learning by Bootstrapping the Covariate Shift (2019, neutral): “The Bellman operator describes a single step of dynamic programming (Bellman 1957) or bootstrap; the process V k+1 := TπV k converges to V π .”
Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation (2018, neutral): “Analogous to the well-known “curse of dimensionality” in dynamic programming [2], we call this problem the “curse of horizon” in off-policy learning.”
Safe Policy Improvement with Baseline Bootstrapping (2017, positive): “In practice, it may be achieved in a model-based manner by explicitly computing the MDP model (cid:99) M , constructing the set of allowed policies Π b and ﬁnally searching for the Π b -optimal policy π spibb in (cid:99) M using policy iteration over Π b (Howard, 1966; Puter-man & Brumelle, 1979).”
The Uncertainty Bellman Equation and Exploration (2017, positive): “One inefﬁciency in these algorithms is that, although the concentration may be tight at each state and action independently, the combination of simultaneously optimistic estimates may result in an extremely over-optimistic estimate for the MDP as a whole [28].”
Distributional Reinforcement Learning with Quantile Regression (2017, neutral): “Because knowing this value for the optimal policy is sufﬁcient to act optimally, it is the object modelled by classic value-based meth-ods such as SARSA (Rummery and Niranjan 1994) and Q-Learning (Watkins and Dayan 1992), which use Bellman’s equation (Bellman 1957) to efﬁciently reason about value.”
Continuous Doubly Constrained Batch Reinforcement Learning (2021, positive): “In this case, πφ does not simply concentrate on actions that maximize Qθ, so Step 6 no longer implements the Bellman-optimality operator even with M = 1, η = 0.”
Implicit Quantile Networks for Distributional Reinforcement Learning (2018, positive): “One approach is to find the unique fixed point Q∗ = Q ∗ of the Bellman optimality operator (Bellman, 1957):”
A Distributional Perspective on Reinforcement Learning (2017, neutral): “In this paper we take away the expectations inside Bell-man’s equations and consider instead the full distribution of the random variable Z π .”
Multiagent Learning: Basics, Challenges, and Prospects (2012, neutral): “The value for a given policy π, expressed by equation 1, can iteratively be computed by the Bellman Equation (Bellman 1957).”
Successor Features for Transfer in Reinforcement Learning (2016, positive): “The ﬁrst one is a generalization of Bellman’s [4] classic policy improvement theorem that extends the original result from one to multiple decision policies.”
Benchmarking Batch Deep Reinforcement Learning Algorithms (2019, neutral): “For γ ∈ [0 , 1) the value function and optimal value function are the unique ﬁxed points of the Bellman operator T π and optimality operator T ∗ , respectively [Bellman, 1957, Bertsekas and Tsitsiklis, 1996]: (1) (2) Deep Reinforcement Learning.”