Adaptive Skills Adaptive Partitions (ASAP)

By Daniel Jaymin Mankowitz, Timothy A. Mann, Shie Mannor. 2016

61 citations recorded by the source corpus.

We introduce the Adaptive Skills, Adaptive Partitions (ASAP) framework that (1) learns skills (i.e., temporally extended actions or options) as well as (2) where to apply them. We believe that both (1) and (2) are necessary for a truly general skill learning framework, which is a key building block needed to scale up to lifelong learning agents. The ASAP framework can also solve related new tasks simply by adapting where it applies its existing learned skills. We prove that ASAP converges to a local optimum under natural conditions. Finally, our experimental results, which include a RoboCup domain, demonstrate the ability of ASAP to learn where to reuse skills as well as solve multiple tasks with considerably less experience than solving each task from scratch.

Subfields: Artificial Intelligence, Computer Networks and Communications, Industrial and Manufacturing Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Industrial and Manufacturing Engineering

Citations in the corpus

Graying the black box: Understanding DQNs (2016, background): “First, one must consider the high complexity of the state space, and second, the SMDP model requires to identify a set of skills, a challenging problem with no easy solution (Mankowitz et al., 2016).”
A Deep Hierarchical Approach to Lifelong Learning in Minecraft (2016, background): “Recent work in RL has provided insights into learning reusable skills [Mankowitz et al. , 2016a; Mankowitz et al. , 2016b], but this has been limited to low dimensional problems.”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “Mankowitz, Mann, and Mannor (2016) proposed a framework to learn near-optimal skills for a task, composing these skills together to enable efficient multi-task learning.”
The Option-Critic Architecture (2016, background): “…but approaches that can be used naturally with continuous state and/or action spaces have only recently started to become feasible (Konidaris et al. 2011; Niekum 2013; Mann, Mannor, and Precup 2015; Mankowitz, Mann, and Mannor 2016; Kulkarni et al. 2016; Vezhnevets et al. 2016; Daniel et al. 2016).”
Eigenoption Discovery through the Deep Successor Representation (2017, background): “Option discovery has received ample attention over many years, with varied solutions being proposed (e.g., Bacon et al., 2017; Şimsek & Barto, 2004; Daniel et al., 2016; Florensa et al., 2017; Konidaris & Barto, 2009; Mankowitz et al., 2016; McGovern & Barto, 2001).”
An Introduction to Deep Reinforcement Learning (2018, unknown): “Many variations of these approaches are also of interest (e.g., Kulkarni et al., 2016; Mankowitz et al., 2016).”
Learning Abstract Options (2018, background): “Discovering these temporal abstractions autonomously has been the subject of extensive research [16, 28, 17, 27, 26] with approaches that can be used in continuous state and/or action spaces only recently becoming feasible [9, 20, 15, 14, 10, 31, 3].”
Discovering Options for Exploration by Minimizing Cover Time (2019, background): “, 2002; Konidaris & Barto, 2009), while other approaches use gradient descent to generate options using the observed rewards (Mankowitz et al., 2016; Bacon et al., 2017; Harb et al., 2017).”
A Laplacian Framework for Option Discovery in Reinforcement Learning (2017, methodology): “…to new value functions ( e.g. , McGovern & Barto, 2001; Menache et al., 2002; Konidaris & Barto, 2009), methods that use the observed rewards to climb a gradient ( e.g. , Mankowitz et al., 2016; Vezhnevets et al., 2016; Bacon et al., 2017), or to do probabilistic inference (Daniel et al., 2016).”
Learning Robust Options (2018, methodology): “In the linear setting we show that the non-robust version of a linear option learning algorithm called ASAP (Mankowitz, Mann, and Mannor 2016a), learns an inherently robust set of options.”
Context-Specific Representation Abstraction for Deep Option Learning (2021, methodology): “Similar to option discovery methods (Mankowitz, Mann, and Mannor 2016; Daniel et al. 2016; Vezhnevets et al. 2016), we also assume that all options are available in each state.”
When Waiting is not an Option : Learning Options with a Deliberation Cost (2017, background): “…abstraction in reinforcement learning has increased substantially in the last couple of years, due to a series of success in constructing such abstractions automatically from data, e.g. (Bacon et al. 2017; Kulkarni et al. 2016; Daniel et al. 2016; Mankowitz et al. 2016; Machado et al. 2017).”
Spatio-Temporal Abstractions in Reinforcement Learning Through Neural Encoding (2016, background): “First, one must consider the high complexity of the state space, and second, the SMDP model requires to identify a set of skills, a challenging problem with no easy solution (Mankowitz et al., 2016).”
Unicorn: Continual Learning with a Universal, Off-policy Agent (2018, background): “…can be framed as a higher level decision problem in a hierarchical setup; and the literature on hierarchical RL contains numerous approaches that would be compatible with our setup [42, 3, 26, 48, 23]; possibly allowing the higher-level agent to switch goal signals more often than once per episode.”
An empirical investigation of the challenges of real-world reinforcement learning (2020, methodology): “Previous research in this area includes learning the options from scratch [Mankowitz et al. 2016a;b; Bacon et al. 2017] as well as planning, given a pre-trained set of options [Schaul et al. 2015; Mankowitz et al. 2018b].”
Challenges of real-world reinforcement learning: definitions, benchmarks and analysis (2021, background): “This return-equivalent MDP is easier to learn, is guaranteed to have the same optimal policy as the original MDP, and the approach shows improvements in Atari tasks with long delays.”