State abstraction for programmable reinforcement learning agents

By David Andre, Stuart J. Russell. 2002

315 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Research Priorities for Robust and Beneficial Artificial Intelligence (2015, background): “Some work has been done on verifying neural network applications [35–37] and the notion of partial programs [38, 39] allows the designer to impose arbitrary “structural” constraints on behavior, but much remains to be done before it will be possible to have high conﬁdence that a learning agent will…”
Representation Matters: Offline Pretraining for Sequential Decision Making (2021, background): “Abstraction and Bisimulation Traditionally, representation learning has been framed as learning or identifying abstractions of the state or action space of an environment (Andre and Russell, 2002; Mannor et al., 2004; Dearden and Boutilier, 1997; Abel et al., 2018).”
Modular Multitask Reinforcement Learning with Policy Sketches (2016, methodology): “Like our approach, HAMs begin with a representation of a high-level policy as an automaton (or a more general computer program; Andre & Russell, 2001; Marthi et al., 2004) and use reinforcement learning to ﬁll in low-level details.”
Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning (2017, background): “Our work is also closely related to Programmable HAMs (PHAMs) (Andre and Russell, 2000; 2002) in that a PHAM is designed to execute a given program.”
Learning and Transfer of Modulated Locomotor Controllers (2016, background): “These temporal abstractions may be applied to motor control [5, 26, 29], may be transferred to new tasks [1, 2, 21], and may also incorporate information hiding principles [6, 14].”
Near Optimal Behavior via Approximate State Abstraction (2016, background): “Thus, a key research agenda for planning and RL is leveraging abstraction to reduce large state spaces [2, 21, 10, 12, 6].”
Inventing Relational State and Action Abstractions for Effective and Efficient Bilevel Planning (2022, background): “Our work continues a long line of research on learning state abstractions [2, 3, 4, 5, 6, 46, 47, 48, 49, 50] and action abstractions [27, 51, 52] for planning.”
Transfer in variable-reward hierarchical reinforcement learning (2008, background): “Hierarchical Reinforcement Learning (HRL) makes it possible to scale RL to large tasks by decomposing them into multiple smaller subtasks (Dietterich 2000; Sutton et al. 1999; Mach Learn (2008) 73: 289–312 291 Andre and Russell 2002 ).”
Hierarchically organized behavior and its neural foundations: A reinforcement-learning perspective (2008, methodology): “Whereas the options framework, as we have discussed, maintains option-specific value functions, some other frameworks, including MAXQ ( Andre & Russell, 2001, 2002; Dietterich, 2000), decompose these value functions into separate components, giving rise to slightly different learning algorithms.”
Multiobjective Reinforcement Learning: A Comprehensive Overview (2015, methodology): “There have been some HRL approaches or formulations to incorporate prior knowledge: HAMs [81], MAXQ [82], options [83], and ALisp [84].”
On Reinforcement Learning for Full-length Game of StarCraft (2018, methodology): “r method achieves state-of-the-art results. 2 2 Background 2.1 Reinforcement Learning Consider a ﬁnite-horizon Markov Decision Process (MDP), which can be speciﬁed as 6-tuple: M= hS;A;P(:);R(:); ;Ti; (1) Sis the state space and s2Sis a state of the state space. Ais the action space and a2A is an act”
Automated Planning and Acting (2016, background): “Few partial programming languages and systems have been proposed (for example, Alisp [21, 465] or A2BL [542]) and demonstrated in simulations and video games.”
Black-Box Policy Search with Probabilistic Programs (2015, methodology): “In contrast to languages that target a single domainspecific algorithm [Andre and Russell, 2002, Srivastava et al., 2014, Nitti et al., 2015], our formulation emphasizes the use of general-purpose techniques for Bayesian inference, in which learning is used for inference amortization.”
Using Bisimulation for Policy Transfer in MDPs (2010, background): “Previous work has also illustrated the fact that using temporally extended actions (and their models) can significantly improve knowledge transfer in MDPs (e.g., Perkins & Precup, 1999, Andre & Russell, 2002; Ravindran & Barto, 2003; Konidaris & Barto, 2007).”
State Abstractions for Lifelong Reinforcement Learning (2018, background): “…on two forms of abstraction: (1) State abstraction, which groups together similar world-states to form compressed descriptions of the environment (Andre & Russell, 2002; Jong & Stone, 2005; Dietterich, 2000), and (2) Action abstraction, which yields compact models of temporally extended behavior…”