TD Models: Modeling the World at a Mixture of Time Scales

By Richard S. Sutton. 1995

Subfields: Reinforcement Learning, Time-Series & Forecasting

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Time-Series & Forecasting

Citations in the corpus

On Convergence of Emphatic Temporal-Difference Learning (2015, methodology): “For the details of this Bellman equation, we refer the readers to the early work (Sutton, 1995; Sutton and Barto, 1998) and the recent work (Sutton et al., 2015).”
An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learning (2015, methodology): “ng function : S ![0;1] specifying a potentially dierent degree of bootstrapping, 1 (s), for each state s. General bootstrapping of this form has been partially developed in several previous works (Sutton 1995, Sutton & Barto 1998, Maei & Sutton 2010, Sutton et al. 2014, cf. Yu 2012). As a no”
FeUdal Networks for Hierarchical Reinforcement Learning (2017, neutral): “…learning (Sutton et al., 1999; Precup, 2000; Dayan & Hinton, 1993; Dietterich, 2000; Boutilier et al., 1997; Dayan, 1993; Kaelbling, 2014; Parr & Russell, 1998; Precup et al., 1997; 1998; Schmidhuber, 1991; Sutton, 1995; Wiering & Schmidhuber, 1997; Vezhnevets et al., 2016; Bacon et al., 2017).”
Hierarchical Reinforcement Learning with the MAXQ Value Function Decomposition (1999, background): “In a series of papers, Sutton and his colleagues (Sutton, 1995; Precup & Sutton, 1998; Sutton et al., 1998) have studied a kind of subtask that they call an \option.”
Strategic Attentive Writer for Learning Macro-Actions (2016, background): “2 Related Work Learning temporally extended actions and temporal abstraction in general are long standing problems in reinforcement learning [5, 6, 7, 8, 12, 20, 21, 22, 23, 25, 27, 28, 30].”
Emphatic Temporal-Difference Learning (2015, neutral): “…ETD(λ) is a multistep, generalized Bellman equation which is determined by the bootstrapping parameters λ(s) and also has vπ as its unique solution (Sutton 1995): v = rλπ + P λ πv, (14) where Pλπ is a substochastic matrix and r λ π ∈ RN .2 Let Φ be the N × n matrix with the feature vectors…”
Reinforcement Learning with Replacing Eligibility Traces (2005, background): “Arguably, yet a third mechanism for managing delayed reward is to change representations orworld models (e.g., Dayan, 1993 ; Sutton, 1995).2.”
Hierarchical Solution of Markov Decision Processes using Macro-actions (1998, methodology): “an backup. After some finite number n of iterations, the choice of maximizing ac tion for each s forms an optimal policy 7r and vn approxi mates its value. 2.2 Macro-actions and their Models Sutton [23] has argued that it is crucial to be able to model MDPs at multiple time scales. The ability to de”
Unifying Task Specification in Reinforcement Learning (2016, methodology): “(2)A generalization to state-based trace parameters has been considered (Sutton, 1995; Sutton and Barto, 1998b; Maei and Sutton, 2010; Sutton et al., 2014; Yu, 2012).”
Multi-timescale nexting in a reinforcement learning robot (2011, methodology): “The prior work pursuing this approach has almost all been in simulation, and has used table-lookup representations and a small number of predictions (e.g., Sutton 1995, Kaelbling 1993, Singh 1992, Sutton, Precup & Singh 1999, Dayan and Hinton 1993).”
Intra-Option Learning about Temporally Abstract Actions (1998, methodology): “Others may be possible using eligibility traces and standard tricks for off-policy learning (see Sutton, 1995; Sutton and Barto, 1998).”
Reinforcement Learning: An Introduction (1998, methodology): “11.3 The deadly triad was first identified by Sutton (1995b) and thoroughly analyzed by Tsitsiklis and Van Roy (1997).”
Compositional Planning Using Optimal Option Models (2012, background): “Several of these dimensions have been partially explored by prior work.”
Strengths, Weaknesses, and Combinations of Model-based and Model-free Reinforcement Learning (2015, methodology): “We prove that the solution found by linear TD(λ) is identical to the linear β-model solution. β-model (Sutton, 1995) is a mixture of multi-step models in which the weighting falls off exponentially with time: r β = ∞ t=0 (γβP) t r = (I − γβP) −1 r and (3.13) P β = (1 − β) ∞ t=0 (γβP) t P .”
Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning (1999, methodology): “Others may be possible using eligibility traces and standard tricks for off-policy learning (as in [71]).”