Multi-Level Discovery of Deep Options

By Roy Fox, Sanjay Krishnan, Ion Stoica, Ken Goldberg. 2017

132 citations recorded by the source corpus.

Augmenting an agent's control with useful higher-level behaviors called options can greatly reduce the sample complexity of reinforcement learning, but manually designing options is infeasible in high-dimensional and abstract state spaces. While recent work has proposed several techniques for automated option discovery, they do not scale to multi-level hierarchies and to expressive representations such as deep networks. We present Discovery of Deep Options (DDO), a policy-gradient algorithm that discovers parametrized options from a set of demonstration trajectories, and can be used recursively to discover additional levels of the hierarchy. The scalability of our approach to multi-level hierarchies stems from the decoupling of low-level option discovery from high-level meta-control policy learning, facilitated by under-parametrization of the high level. We demonstrate that using the discovered options to augment the action space of Deep Q-Network agents can accelerate learning by guiding exploration in tasks where random actions are unlikely to reach valuable states. We show that DDO is effective in adding options that accelerate learning in 4 out of 5 Atari RAM environments chosen in our experiments. We also show that DDO can discover structure in robot-assisted surgical videos and kinematics that match expert annotation with 72% accuracy.

Subfields: Artificial Intelligence, Management Science and Operations Research, Ocean Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Ocean Engineering

Citations in the corpus

Learning World Graphs to Accelerate Hierarchical Reinforcement Learning (2019, neutral): “Pivotal state discovery is related to unsupervised sequence segmentation [17, 47, 76, 13, 16] and option or sub-task discovery in the context of RL [48, 6, 72, 30, 56, 53, 52].”
Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning (2019, methodology): “This question is one focus area of hierarchical imitation learning (HIL), where solutions [8, 9] typically try to achieve two goals: i) learn a temporal task abstraction, and ii) discover a meaningful segmentation of the demonstrations into subtasks.”
Skill Induction and Planning with Latent Language (2021, background): “, 2016; Dayan and Hinton, 1992); or performed unsupervised discovery of a finite skill inventory using clustering techniques (Dietterich, 1999; Fox et al., 2017).”
Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddings (2018, neutral): “Some end-to-end gradient-based methods to learn options have recently been proposed as well (Bacon et al., 2017; Fox et al., 2017).”
Search on the Replay Buffer: Bridging Planning and Reinforcement Learning (2019, background): “Another class of algorithms (Drummond, 2002; Fox et al., 2017; S¸ims¸ek et al., 2005) focus solely on automatically discovering these skills or subgoals.”
Idiosyncrasies and challenges of data driven learning in electronic trading (2018, background): “We also note the progress in the end-to-end (rule-free) hierarchical RL where the temporal abstractive property of meta-policy emerges from behaviour or goal clustering by primitive policies [Bacon et al., 2017][Fox et al., 2017][Vezhnevets et al., 2017].”
Variational End-to-End Navigation and Localization (2018, unknown): “Our work also relates to several research efforts in reinforcement learning in subfields such as bridging different levels of planning hierarchies [1], [20], and relating to maps as agents plan and act [21], [22].”
Exploiting Hierarchy for Learning and Transfer in KL-regularized RL (2019, methodology): “This form can be convenient when π 0 ( a t | x t ) and π ( a t | x t ) are tractable (e.g. when z or y have a small number of discrete states or decompose conveniently over time, e.g. as in Fox et al. 2017; Krishnan et al. 2017).”
Compositional Imitation Learning: Explaining and executing one task at a time (2018, methodology): “Our framework is closely related to option discovery [16, 17, 18, 19], with the main difference being that our inference algorithm is agnostic to what type of option (sub-task) encoding is used.”
Directed-Info GAIL: Learning Hierarchical Policies from Unsegmented Demonstrations using Directed Information (2018, background): “Further work on discovery of deep continuous options Krishnan et al. (2017) allows the option policy to also select a continuous action in states where none of the options are applicable.”
Modeling Long-horizon Tasks as Sequential Interaction Landscapes (2020, background): “To address this issue, there has a been a number of hierarchical imitation learning approaches [16, 53, 14, 29] that focus on segmenting long-horizon tasks into subcomponents.”
Multi-Modal Imitation Learning from Unstructured Demonstrations using Generative Adversarial Nets (2017, unknown): “Examples include [23, 18, 14, 31, 13].”
Learning Abstract Options (2018, background): “[5] successfully trained a hierarchy of options end to end for imitation learning.”
Learning Robot Skills with Temporal Variational Inference (2020, methodology): “Our choice of VI over the forward-backward style algorithm employed in Fox et al. (2017); Krishnan et al. (2017) is because VI is amenable to both continuous and discrete latent variables z.”
Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills (2020, background): “These include learning options while solving the desired task (Bacon et al., 2017), leveraging demonstrations (Fox et al., 2017), training goal-oriented low-level policies (Nachum et al., 2018), and meta-learning primitives from a distribution of related tasks (Frans et al., 2018).”
DDCO: Discovery of Deep Continuous Options for Robot Learning from Demonstrations (2017, methodology): “Extending similar concepts to image data has been a recent area of interest, mostly in reinforcement and self-supervision settings for simulated domains [29, 11, 30, 10].”
Variational Option Discovery Algorithms (2018, unknown): “Bacon et al. [2017] and Fox et al. [2017] derive policy gradient methods for learning options: Bacon et al. [2017] learn options concurrently with solving a particular task, while Fox et al. [2017] learn options from demonstrations to accelerate speciﬁc-task learning.”
From Motor Control to Team Play in Simulated Humanoid Football (2021, background): “Furthermore, the behaviour to be modeled can be copied from demonstrations (33, 124, 127, 131), learned as part of a multi-task framework (e.”
CompILE: Compositional Imitation Learning and Execution (2018, methodology): “Fox et al. (2017) introduce an EM-based inference algorithm for option discovery in settings similar to ours, however limited to discrete latent variables and to inference networks that are independent of the position of task boundaries: in their case without recurrency and only dependent on the…”