Multi-time Models for Temporally Abstract Planning

By Doina Precup, Richard S. Sutton. 1997

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Deep learning in neural networks: An overview (2014, methodology): “Numerous alternative HRL techniques have been proposed (e.g., Ring, 1991, 1994; Jameson, 1991; Tenenberg et al., 1993; Moore and Atkeson, 1995; Precup et al., 1998; Dietterich, 2000b; Menache et al., 2002; Doya et al., 2002; Ghavamzadeh and Mahadevan, 2003; Barto and Mahadevan, 2003; Samejima et al.”
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (2015, methodology): “Numerous alternative HRL techniques have been proposed [204, 206, 117, 279, 295, 171, 195, 50, 162, 51, 15, 215, 11, 260].”
Intra-Option Learning about Temporally Abstract Actions (1998, methodology): “…Markov decision processes (MDPs) (e.g., Singh, 1992a,b; Kaelbling, 1993; Lin, 1993; Dayan & Hin-ton, 1993; Thrun and Schwartz, 1995; Sutton, 1995; Hu-ber and Grupen, 1997; Kalm´ar, Szepesv´ari, and L¨orincz, 1997; Dietterich, 1998; Parr and Russell, 1998; Precup, Sutton, and Singh 1997, 1998a,b).”
Spectrum-Diverse Neuroevolution With Unified Neural Models (2017, background): “The importance of time scales has been shown in many articles, for example, in self-localization of a neuroevolved robot in a corridor (the sensory-motor patterns in different time scales are used) [20], in reinforcement learning with multimodels and subgoals [21], with the long short-term memory re”
Adaptive Skills Adaptive Partitions (ASAP) (2016, background): “Model misspecification is defined as having an unsatisfactory set of skills and inter-skill policy that provide a sub-optimal solution to a given task.”
Flexible Decomposition Algorithms for Weakly Coupled Markov Decision Problems (1998, background): “Figure 1 shows a simple navigation MDP di vided into four rooms, each of which can be considered a subproblem.”
Using Options for Knowledge Transfer in Reinforcement Learning (1999, unknown): “Temporally extended actions have been extensively studied in recent work as a powerful tool for generalization in reinforcement learning (e.g. Singh, 1992a,b; Kaelbling, 1993a; Lin, 1993; Dayan & Hinton, 1993; Thrun & Schwartz, 1995; Dietterich, 1998; Parr & Russell, 1998; Hauskrecht et.al, 1998; Ma”
Approximate Value Iteration with Temporally Extended Actions (2015, background): “…well as a wide range of temporally extended actions, and many of the well-known properties of Markov Decision Processes generalize when arbitrary options are added (e.g., Value Iteration and Policy Iteration still converge, Precup & Sutton, 1997; Precup, Sutton, & Singh, 1998; Sutton et al., 1999).”
MDPs: Learning in Varying Environments (2002, background): “…modular reinforcement learning (Mahadevan and Connell, 1992, Singh, 1992, Dayan and Hinton, 1993, Kaelbling, 1993, Mataric, 1997, Kalmár et al., 1998, Dietterich, 2000) and by the formulation of options within a semi-Markov Decision Process framework (Precup and Sutton, 1998, Sutton et al., 1998).”
Value Preserving State-Action Abstractions (2020, methodology): “Similar Models: In each sφ, there is at least one option o that has a similar multi-time model (Precup and Sutton, 1998) to o∗sφ .”
Hierarchical control and learning for markov decision processes (1998, methodology): “Similar representations are used in Precup and Sutton (1997) and Hauskrecht, Meuleau, Boutilier, Kaelbling, and Dean (1998).”
Combating the Compounding-Error Problem with a Multi-step Model (2019, background): “Similarly, option models are multi-step models that are conditioned on one speciﬁc policy and a termination condition [Precup and Sutton, 1998; Sutton et al. , 1999; Silver and Ciosek, 2012].”
Reinforcement Learning with Hierarchies of Machines (1997, result): “Sutton [19] proposes temporal abstractions, which concatenate sequences of state transitions together to permit reasoning about temporally extended events, and which can thereby form a behavioral hierarchy as in [14] and [ 15 ].”