Issues in Using Function Approximation for Reinforcement Learning

By Sebastian Thrun, Anton Schwartz. 1999

Subfields: Reinforcement Learning, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Statistical Learning Theory

Citations in the corpus

Addressing Function Approximation Error in Actor-Critic Methods (2018, background): “Overestimation bias is a property of Q-learning in which the maximization of a noisy value estimate induces a consistent overestimation (Thrun & Schwartz, 1993).”
Taming the Noise in Reinforcement Learning via Soft Updates (2015, background): “for Q instead of a table, even in the absence of transition or cost noise, because the approximation itself introduces noise [18].”
Surprising Negative Results for Generative Adversarial Tree Search (2018, background): “In addition to this bias, there are statistical biases due to limited network capacity, optimization, model mismatch, and max operator or the choice of a [46].”
Deep Reinforcement Learning with Double Q-Learning (2015, methodology): “The result by Thrun and Schwartz (1993) cited above gives an upper bound to the overestimation for a speciﬁc setup, but it is also possible, and potentially more interesting, to derive a lower bound.”
Off-Policy Deep Reinforcement Learning without Exploration (2018, background): “When combined with maximization in reinforcement learning algorithms, extrapolation error provides a source of noise that can induce a persistent overestimation bias (Thrun & Schwartz, 1993; Van Hasselt et al., 2016; Fujimoto et al., 2018).”
Self-Supervised Deep Reinforcement Learning with Generalized Computation Graphs for Robot Navigation (2017, background): “Specifically, for (N -step) Q-learning, bias from bootstrapping and high variance multi-step returns can lead to slow convergence [27].”
Continuous Doubly Constrained Batch Reinforcement Learning (2021, methodology): “OE will be strictly positive when the estimation errors are weakly correlated and will grow with the ER-variance σ [32, 58].”
Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics (2020, background): “Thrun & Schwartz (1993) elucidate the overestimation as a consequence of Jensen’s inequality: the maximum of the Q-function over actions is not greater than the expected maximum of noisy (approximate) Q-function.”
Benchmarking Batch Deep Reinforcement Learning Algorithms (2019, methodology): “This erroneous extrapolation is propagated through temporal difference update of most off-policy algorithms [Sutton, 1988], causing extreme overestimation and poor performance [Thrun and Schwartz, 1993].”
Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning (2016, methodology): “To analyse the TAE variance we ﬁrst must assume a statistical model on the TAE, and we do so in a similar way to Thrun & Schwartz (1993).”
Learning to Play in a Day: Faster Deep Reinforcement Learning by Optimality Tightening (2016, methodology): “However, we are still far from building intelligent solutions for many real-world challenges, such as autonomous driving, human-computer interaction and automated decision making, in which software agents need to consider interactions with a dynamic environment and take actions towards goals.”
Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning (2017, methodology): “r doing this is that van Hasselt et al. (2015) showed that the double DQN algorithm improved the performance of DQN in Atari 2600 games by reducing the overestimation ofthe action values. It isknown (Thrun and Schwartz, 1993; van Hasselt, 2010) that Q-learning based algorithms, such as DQN, can over”
An Intelligent Battery Controller Using Bias-Corrected Q-learning (2012, unknown): “The effect of this particular bias in applications has been observed earlier in estimating asset values in finance (Brown 1974), and by authors in reinforcement learning (Thurn and Schwartz 1993; Ormoneit and Sen 2002), management science (Smith and Winkler 2006) and in operations research (Powell 2”
Mixing Update Q-value for Deep Reinforcement Learning (2019, background): “Overestimation bias is a property of Q-learning in which the maximization of a noisy value estimate induces a consistent overestimation [4], [5].”
Randomized Ensembled Double Q-Learning: Learning Fast Without a Model (2021, methodology): “Similar to Thrun & Schwartz (1993) and Lan et al. (2020), deﬁne the post-update estimation bias as the difference between the target (1) and the target when using the ground-truth: Here we write Z M,N to emphasize its dependence on both M and N .”
Particle swarm optimization for generating interpretable fuzzy reinforcement learning policies (2016, methodology): “orithm in iteration Kby evaluating ˇ[x K](s) 2 argmax a Q[x K](s;a). In practice function approximation in RL is not known to converge to a point [13] and is prone to overestimation of utility values [34]. Since neural networks are not averagers, additional problems are likely to occur. To cover the”
Learn What Not to Learn: Action Elimination with Deep Reinforcement Learning (2018, background): “Function Approximation: It is well known that errors in the Q-function estimates may cause the learning algorithm to converge to a suboptimal policy, a phenomenon that becomes more noticeable in environments with large action spaces (Thrun and Schwartz, 1993).”
Reinforcement Learning with Decision Trees (2003, background): “Thrun and Schwartz [3] discuss another problem with using back-propagation networks.”
Maxmin Q-learning: Controlling the Estimation Bias of Q-learning (2020, methodology): “…stated above and assume all actions share the same true action-value, (i) the expected estimation bias is where Theorem 1 is a generalization of the ﬁrst lemma in Thrun & Schwartz (1993); we provide the proof in Appendix A as well as a visualization of the expected bias for varying M and N .”