Neuro-Dynamic Programming

By Dimitri P. Bertsekas. 2009

4,030 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Ocean Engineering, Reinforcement Learning, Mathematical Optimization

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Ocean Engineering

In Reinforcement Learning

In Mathematical Optimization

Citations in the corpus

Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergence (2020, methodology): “For non-convex functions, such an analysis can be found in Ghadimi and Lan (2013); Bottou et al. (2018). Using a constant step-size for SGD guarantees convergence to a neighbourhoood of the solution.”
IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures (2018, background): “…the usual Robbins-Munro conditions: for each state x , The proof is a straightforward application of the convergence result for stochastic approximation algorithms to the ﬁxed point of a contraction operator, see e.g. Dayan & Sejnowski (1994); Bertsekas & Tsitsiklis (1996); Kushner & Yin (2003).”
Safe and Efficient Off-Policy Reinforcement Learning (2016, background): “The proof extends similar convergence proofs of TD( λ ) by Bertsekas and Tsitsiklis (1996) and of optimistic policy iteration by Tsitsiklis (2003), and is provided in the appendix.”
Unifying Count-Based Exploration and Intrinsic Motivation (2016, methodology): “For α n = n − 2 / 3 , we may appeal to well-known results from stochastic approximation (e.g. Bertsekas and Tsitsiklis, 1996) and ﬁnd that almost surely , we may think of Assumption 1(b) as also requiring ρ to converge at a rate of Θ(1 /n ) for a comparison with the empirical count N n to be…”
Deep learning in neural networks: An overview (2014, background): “2 Deep FNNs for Traditional RL and Markov Decision Processes (MDPs) The classical approach to RL (Samuel, 1959; Bertsekas and Tsitsiklis, 1996; Sutton and Barto, 1998) makes the simplifying assumption of Markov Decision Processes (MDPs): the current input of the RL agent conveys all information nece”
RUDDER: Return Decomposition for Delayed Rewards (2018, background): “0 since after some time the agent does no longer receive reward in MDPs with ﬁnite time horizon or in MDPs with absorbing states that have zero reward. Tˇ is monotonically increasing in its arguments [14]. For q1 and q2 with the component-wise condition q1 > q2, we have Tˇ[q 1](s;a) Tˇ[q2](s;a) (”
Meta-Gradient Reinforcement Learning (2018, background): “ frequently observed that discounts <1 achieve better results [Prokhorov and Wunsch, 1997], especially during early learning. It is known that many algorithms converge faster with lower discounts [Bertsekas and Tsitsiklis, 1996], but of course too low a discount can lead to highly sub-optimal pol”
Off-Policy Deep Reinforcement Learning without Exploration (2018, methodology): “For a given policy π , the value function can be estimated using sampled versions of the Bellman operator T π : (1) The Bellman operator is a contraction for γ ∈ [0 , 1) with unique ﬁxed point Q π ( s, a ) (Bertsekas & Tsitsiklis, 1996).”
Doubly Robust Off-policy Value Evaluation for Reinforcement Learning (2015, methodology): “To generalize, one can either apply function approximation to ﬁtting (cid:99) M (Jong & Stone, 2007; Gr¨unew¨alder et al., 2012), or to ﬁtting the value function directly (Bertsekas & Tsitsiklis, 1996; Sutton & Barto, 1998; Dann et al., 2014).”
MOReL : Model-Based Offline Reinforcement Learning (2020, background): “Furthermore, MBRL can draw upon the rich literature on model-based planning including model predictive control (MPC) [23, 24, 61, 62], search based planning [25, 63], dynamic programming [26, 64], and policy optimization [32, 65, 33, 27, 53].”
Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning (2020, background): “In particular, several papers have theoretically analyzed the accumulation of bootstrapping errors in approximate dynamic programming (Bertsekas & Tsitsiklis, 1996; Munos, 2005) and approximate policy iteration (Farahmand et al., 2010; Scherrer et al., 2015); for the latter of which there exist…”
Stochastic Primal-Dual Q-Learning (2018, neutral): “Reinforcement learning (RL) (Sutton and Barto, 1998; Bertsekas and Tsitsiklis, 1996; Puterman, 2014) is a subﬁeld of machine learning which addresses the problem of how an autonomous agent (decision maker) can learn an optimal policy to maximize long-term cumulative rewards, while inter-acting with…”
Boosting the Actor with Dual Critic (2017, methodology): “However, the relation between the quality of the learned value function and the quality of the derived policy is fairly weak (Bertsekas and Tsitsiklis, 1996).”