Generalization in Reinforcement Learning: Safely Approximating the Value Function

By Justin A. Boyan, Andrew W. Moore. 1994

Subfields: Reinforcement Learning, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Scaling & Learning Theory

Citations in the corpus

Fitted Q-Learning for Relational Domains (2020, methodology): “A set of classical approaches for this problem based on Approximate Dynamic Programming (ADP) is the ﬁtted value iteration algorithm (Boyan and Moore 1995; Ernst, Geurts, and Wehenkel 2005; Ried-miller 2005), a batch mode approximation scheme that employs function approximators in each iteration to…”
Reinforcement learning in robotics: A survey (2013, background): “While this may lead more quickly to good, possibly globally optimal solutions, such learning processes often prove unstable under function approximation (Boyan and Moore, 1995; Kakade and Langford, 2002; Bagnell et al., 2003) and are considerably more dangerous when applied to real systems where ove”
Striving for Simplicity in Off-policy Deep Reinforcement Learning (2019, background): “For instance, it is well known that Q-learning can be unstable or even divergent when neural networks are used to parameterize optimal action-value functions [2, 9, 58].”
Smoothed Dual Embedding Control (2017, methodology): “proof typically relies on the contraction property (Bertsekas and Tsitsiklis, 1996). Unfortunately, when function approximatiors are used, such xed-point methods can easily become unstable/divergent (Boyan and Moore, 1995; Baird, 1995; Tsitsiklis and Van Roy, 1997), except in rather limited cases. F”
Reinforcement Learning: A Survey (1996, methodology): “These issues are discussed by Boyan and Moore (1995), who give some simple examples of value function errors growing arbitrarily large when generalization is used with value iteration.”
Batch Mode Reinforcement Learning based on the Synthesis of Artificial Trajectories (2012, background): “The same behavior could probably be observed by using other algorithms that combine dynamic programming strategies with kernel-based approximators or averagers (Boyan and Moore 1995; Gordon 1999; Ormoneit and Sen 2002).”
A survey and critique of multiagent deep reinforcement learning (2018, methodology): “for example, decision trees [66], tile coding [67], radial basis functions [68], and locally weighted regression [69] to approximate the value function.”
Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning (2016, background): “Except for few successful attempts (e.g., TD-gammon, Tesauro (1995)), the combination of non-linear function approximation and RL was considered unstable and was shown to diverge even in simple domains (Boyan & Moore, 1995).”
A Kernel Loss for Solving the Bellman Equation (2019, methodology): “…such as neural networks is used to represent the value function in large-scale problems, the critical property of contraction is generally lost (e.g., Boyan & Moore, 1995; Baird, 1995; Tsitsiklis & Van Roy, 1997), except in rather restricted cases (e.g., Gordon, 1995; Tsitsiklis & Van Roy, 1997).”
Generating Legible Motion (2013, methodology): “In such cases, finding good approximations for V becomes crucial, many techniques value function approximation techniques can be applied toward this goal [8].”
Algorithms for Reinforcement Learning (2010, background): “It is known that fitted Q-iteration might diverge unless a special regressor is used (Baird, 1995; Boyan and Moore, 1995; Tsitsiklis and Van Roy, 1996).”
Tree based hierarchical reinforcement learning (2002, methodology): “Boyan and Moore (1995, 1996) noted that if performed offline, these ’unstable’ approximation algorithms could be made stable. This culminated in the Batch Fit to Best Paths (BFBP) algorithm of Wang and Dietterich (2002). This the same overall approach used in our thesis.”
Multidimensional Triangulation and Interpolation for Reinforcement Learning (1996, unknown): “[Boyan and Moore, 1995].”
SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation (2017, methodology): “Unfortunately, when function approximatiors are used, such ﬁxed-point methods easily become unstable or even divergent (Boyan & Moore, 1995; Baird, 1995; Tsitsiklis & Van Roy, 1997), except in a few special cases.”
Stable Fitted Reinforcement Learning (1995, unknown): “G.J.”
Adaptive Tile Coding for Value Function Approximation (2007, background): “In the mountain car task ( Boyan & Moore 1995 ), depicted in Figure 4, an agent must drive a car to the top of a steep mountain.”
Kernel-Based Reinforcement Learning (2004, background): “For a detailed discussion of this problem, as well as a list of exceptions, the interested reader is referred to Boyan and Moore (1995) and Tsitsiklis and Van Roy (1996). By adopting a non-parametric perspective on reinforcement learning, we suggest an algorithm that always converges to a unique sol”