Generalization in Reinforcement Learning: Successful Examples Using Sparse Coarse Coding

By Richard S. Sutton. 1995

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Investigating Contingency Awareness Using Atari 2600 Games (2012, methodology): “When using function approximation with popular modelfree reinforcement learning algorithms such as SARSA(λ), it is well known (Sutton 1996; Tesauro 1995) that good performance hinges on having access to an appropriate set of basis functions or features.”
Safe and Efficient Off-Policy Reinforcement Learning (2016, background): “One fundamental trade-off in reinforcement learning lies in the definition of the update target: should one estimate Monte Carlo returns or bootstrap from an existing Q-function? Return-based methods (where return refers to the sum of discounted rewards ∑ t γ rt) offer some advantages over value boo”
Toward Interpretable-AI Policies Using Evolutionary Nonlinear Decision Trees for Discrete-Action Systems (2020, methodology): “In our approach described below, we build an efﬁcient search algorithm to directly ﬁnd relatively interpretable policies as compared to the black-box policies represented using DNN (or tile encoding [12]) using recent advances in nonlinear optimization.”
Reinforcement Learning via Fenchel-Rockafellar Duality (2020, background): “Most of the eﬀective existing RL algorithms are rooted in this dynamic programming paradigm, attempting to ﬁnd approximate ﬁxed-point solutions to the Bellman recurrence, leading to the family of temporal-diﬀerence (TD) algorithms including SARSA (Sutton, 1996), Q -learning (Watkins, 1989), and thei”
Smoothed Dual Embedding Control (2017, background): “The resulting algorithm resembles to SARSA with particular policy.”
Learning to Act by Predicting the Future (2016, methodology): “The convergence of such methods was analyzed early on and they were seen as theoretically advantageous, particularly when function ap-proximators are used (Bertsekas, 1995; Sutton, 1995; Singh & Sutton, 1996).”
A General Family of Robust Stochastic Operators for Reinforcement Learning (2018, neutral): “This problem is ﬁrst discussed in [12].”
Increasing the Action Gap: New Operators for Reinforcement Learning (2015, background): “…to extend the consistent Bellman operator to Q-value approximation schemes which lack a probabilistic interpretation, such as linear approximation (Sutton 1996), locally weighted regression (Atkeson 1991), neural networks (Tesauro 1995), or even information-theoretic methods (Veness et al. 2015).”
Reinforcement Learning: A Survey (1996, neutral): “There has been some recent work on making the updates more e(cid:14)cient (Cichosz & Mulawka, 1995) and on changing the de(cid:12)nition to make TD((cid:21)) more consistent with the certainty-equivalent method (Singh & Sutton, 1996), which is discussed in Section 5.1.”
Count-Based Exploration with Neural Density Models (2017, methodology): “It is well known that better performance, both in terms of learning efﬁciency and approximation error, is attained by multi-step methods (Sutton, 1996; Tsitsiklis & van Roy, 1997).”
Successor Features for Transfer in Reinforcement Learning (2016, background): “In order to do so we introduce a generalized version of a classic RL task known as the “puddle world” [20].”
A Deeper Look at Experience Replay (2017, background): “The initial values for all state-action pairs are set to 0, which is an optimistic initialization (Sutton 1996) to encourage exploration.”
Dyna-Style Planning with Linear Function Approximation and Prioritized Sweeping (2008, background): “The original papers on the Dyna architecture and most subsequent extensions (e.g., Singh 1992; Peng & Williams 1993; Moore & Atkeson 1993; Kuvayev & Sutton 1996) assumed a Markov environment with a tabular representation of states.”
The Effect of Planning Shape on Dyna-style Planning in High-dimensional State Spaces (2018, methodology): “One was based on the Sarsa algorithm [Rummery and Niranjan, 1994; Sutton, 1996], using linear value function approximation with Blob-PROST features [Liang et al.”
A survey and critique of multiagent deep reinforcement learning (2018, neutral): “for example, decision trees [66], tile coding [67], radial basis functions [68], and locally weighted regression [69] to approximate the value function.”
The Utility of Sparse Representations for Control in Reinforcement Learning (2018, background): “In fact, many hand-crafted features are sparse representations, including tile coding (Sutton 1996; Sutton and Barto 1998), radial basis functions and sparse distributed memory (Kanerva 1988; Ratitch and Precup 2004).”
Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning (2017, methodology): “The eligibility trace vector, e t , is for TD( λ ) and for Sarsa( λ ).”