Residual Algorithms: Reinforcement Learning with Function Approximation

By Leemon C. Baird III. 1995

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Playing Atari with Deep Reinforcement Learning (2013, background): “Furthermore, it was shown that combining model-free reinforcement learning algorithms such as Qlearning with non-linear function approximators [25], or indeed with off-policy learning [1] could cause the Q-network to diverge.”
Learning Continuous Control Policies by Stochastic Value Gradients (2015, methodology): “Since SVG(1) is modelbased, we can also use Bellman residual minimization [3].”
Compatible Value Gradients for Reinforcement Learning of Continuous Deep Policies (2015, methodology): “Residual gradient (RG) and gradient temporal difference (GTD) methods were introduced in (Baird, 1995; Sutton et al., 2009a,b).”
When to use parametric models in reinforcement learning? (2019, methodology): “, 2000) can be unstable (Williams and Baird III, 1993; Baird, 1995; Sutton, 1995; Tsitsiklis and Van Roy, 1997; Sutton et al., 2009, 2016) — this is sometimes called the deadly triad (Sutton and Barto, 2018; van Hasselt et al.”
Deep Reinforcement Learning with Double Q-Learning (2015, background): “If seen in isolation, one might perhaps be tempted to think the observed instability is related to inherent instability problems of off-policy learning with function approximation (Baird, 1995; Tsitsiklis and Van Roy, 1997; Sutton et al., 2008; Maei, 2011; Sutton et al., 2015).”
Deep learning in neural networks: An overview (2014, neutral): “Many variants of traditional RL exist (e.g., Abounadi, Bertsekas, & Borkar, 2002; Baird, 1995; Baird & Moore, 1999; Barto, Sutton, & Anderson, 1983; Bertsekas, 2001; Bradtke, Barto, & Kaelbling, 1996; Brafman & Tennenholtz, 2002; Kaelbling, Littman, & Cassandra, 1995; Lagoudakis & Parr, 2003; Maei &”
GenDICE: Generalized Offline Estimation of Stationary Values (2020, background): “Moreover, obtaining unbiased gradients with a naive approach requires double sampling (Baird, 1995).”
GradientDICE: Rethinking Generalized Offline Estimation of Stationary Values (2020, methodology): “This mismatch leads to divergence of the off-policy linear temporal difference learning algorithm (Baird, 1995; Tsitsiklis & Van Roy, 1997). Precup et al. (2001) address this issue with products of importance sampling ratios, which, however, suffer from a large variance.”
AlgaeDICE: Policy Gradient from Arbitrary Experience (2019, methodology): “• As residual learning is known to be hard in function approximation settings (Baird, 1995), we replace ν ( s 0 , a 0 ) in (25) with a mixture η · ν ( s 0 , a 0 ) + (1 − η ) · ν ( s 0 , a 0 ) where ν ( s 0 , a 0 ) is a target value calculated as in Haarnoja et al. (2018).”
Behavior Regularized Offline Reinforcement Learning (2019, methodology): “These failure cases are hypothesized to be caused by erroneous generalization of the state-action value function (Q-value function) learned with function approximators, as suggested by Sutton (1995); Baird (1995); Tsitsiklis & Van Roy (1997); Van Hasselt et al. (2018).”
Striving for Simplicity in Off-policy Deep Reinforcement Learning (2019, background): “Off-policy learning with function approximation has no known convergence guarantees [53], and there are well-known counter examples [2] where Q-learning diverges even with linear function approximation.”
Diagnosing Bottlenecks in Deep Q-learning Algorithms (2019, methodology): “Baird (1995) introduces a simple counterexample in which Watkin’s Q-learning with linear approximators can cause un-Diagnosing We see that for small architectures, there is a signiﬁcant gap be-tween the solution found by FQI (FQI Error) and the best solution within the model class (Project Error).…”
Off-Policy Deep Reinforcement Learning by Bootstrapping the Covariate Shift (2019, background): “However, we cannot recover such an estimate with a single sample s ∼ dμ, in a classic case of the double-sampling problem (Baird 1995).”
Deep Reinforcement Learning and the Deadly Triad (2018, background): “Partial solutions have been proposed (Baird, 1995; Precup and Sutton, 2001; Sutton et al., 2008; Maei et al., 2009; Maei, 2011; Sutton et al., 2016), but they have mostly not been extended to non-linear function approximation, and have not been widely adopted in practice.”
Smoothed Dual Embedding Control (2017, methodology): “on the contraction property (Bertsekas and Tsitsiklis, 1996). Unfortunately, when function approximatiors are used, such xed-point methods can easily become unstable/divergent (Boyan and Moore, 1995; Baird, 1995; Tsitsiklis and Van Roy, 1997), except in rather limited cases. For example, for some ra”
Generalized Emphatic Temporal Difference Learning: Bias-Variance Analysis (2015, methodology): “Indeed, for the TD(0) algorithm (Sutton and Barto, 1998; corresponding to the β = 0 case in our setting), a similar representation as a projected Bellman operator holds, but it may be shown that in the off-policy setting the algorithm might diverge (Baird 1995).”
Emphatic TD Bellman Operator is a Contraction (2015, background): “Indeed, for the TD(0) algorithmSutton & Barto (1998), a similar representation as a projected Bellman operator holds, but it may be shown that in the offpolicy setting the algorithm diverges (Baird, 1995).”
On Convergence of Emphatic Temporal-Difference Learning (2015, background): “See the papers (Baird, 1995; Tsitsiklis and Van Roy, 1997; Sutton et al., 2015) and the books (Bertsekas and Tsitsiklis, 1996; Sutton and Barto, 1998) for related examples and discussion.”
An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learning (2015, methodology): “Simple counter xamples were presented by Baird (1995) and by Tsitsiklis and Van Roy (1997) who also developed a theoretical understanding of the instability.”