DDCO: Discovery of Deep Continuous Options for Robot Learning from Demonstrations

By Sanjay Krishnan, Roy Fox, Ion Stoica, Ken Goldberg. 2017

89 citations recorded by the source corpus.

An option is a short-term skill consisting of a control policy for a specified region of the state space, and a termination condition recognizing leaving that region. In prior work, we proposed an algorithm called Deep Discovery of Options (DDO) to discover options to accelerate reinforcement learning in Atari games. This paper studies an extension to robot imitation learning, called Discovery of Deep Continuous Options (DDCO), where low-level continuous control skills parametrized by deep neural networks are learned from demonstrations. We extend DDO with: (1) a hybrid categorical-continuous distribution model to parametrize high-level policies that can invoke discrete options as well continuous control actions, and (2) a cross-validation method that relaxes DDO's requirement that users specify the number of options to be discovered. We evaluate DDCO in simulation of a 3-link robot in the vertical plane pushing a block with friction and gravity, and in two physical experiments on the da Vinci surgical robot, needle insertion where a needle is grasped and inserted into a silicone tissue phantom, and needle bin picking where needles and pins are grasped from a pile and categorized into bins. In the 3-link arm simulation, results suggest that DDCO can take 3x fewer demonstrations to achieve the same reward compared to a baseline imitation learning approach. In the needle insertion task, DDCO was successful 8/10 times compared to the next most accurate imitation learning baseline 6/10. In the surgical bin picking task, the learned policy successfully grasps a single object in 66 out of 99 attempted grasps, and in all but one case successfully recovered from failed grasps by retrying a second time.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning (2020, methodology): “…2020; Shankar & Gupta, 2020; Krishnan et al., 2017; Merel et al., 2018) which extract skills from a prior dataset and using it for either performing tasks directly (Lynch et al., 2020) or learning downstream tasks (Shankar & Gupta, 2020; Krishnan et al., 2017; Merel et al., 2018) with online RL.”
Diversity is All You Need: Learning Skills without a Reward Function (2018, background): “Previous work on hierarchical RL has learned skills to maximize a single, known, reward function by jointly learning a set of skills and a meta-controller (e.g., (Bacon et al., 2017; Heess et al., 2016; Dayan & Hinton, 1993; Frans et al., 2017; Krishnan et al., 2017; Florensa et al., 2017)).”
Parrot: Data-Driven Behavioral Priors for Reinforcement Learning (2020, background): “…et al., 2019), and methods that learn temporally extended actions, also known as options , from demonstrations, and then recompose them to perform long-horizon tasks through RL or planning (Fox et al., 2017; Krishnan et al., 2017; Kipf et al., 2019; Shankar et al., 2020; Shankar & Gupta, 2020).”
From Motor Control to Team Play in Simulated Humanoid Football (2021, background): “Furthermore, the behaviour to be modeled can be copied from demonstrations (33, 124, 127, 131), learned as part of a multi-task framework (e.”
Discovering Motor Programs by Recomposing Demonstrations (2020, background): “In contrast with most prior work in the options framework, (Daniel et al., 2016; Fox et al., 2017; Krishnan et al., 2017) learn options from a set of demonstrations, rather than in the RL setting.”
Exploiting Hierarchy for Learning and Transfer in KL-regularized RL (2019, methodology): “using the forwardbackward algorithm as in (Fox et al., 2017; Krishnan et al., 2017)); in other cases we can learn a parameterized approximation to the true posterior or can conceivably apply mixed inference schemes (e.”
Directed-Info GAIL: Learning Hierarchical Policies from Unsegmented Demonstrations using Directed Information (2018, background): “Further work on discovery of deep continuous options Krishnan et al. (2017) allows the option policy to also select a continuous action in states where none of the options are applicable.”
CompILE: Compositional Imitation Learning and Execution (2018, methodology): “Our framework is closely related to option discovery (Niekum et al., 2013; Kroemer et al., 2015; Fox et al., 2017; Hausman et al., 2017; Krishnan et al., 2017; Fox et al., 2018), with the main difference being that our inference algorithm is agnostic to what type of option (sub-task) encoding is…”
A Berkeley View of Systems Challenges for AI (2017, background): “In another example, action composition would allow building systems where learning of hierarchical options [63] takes place on powerful clusters in the cloud, and then execution of these options happens at the edge.”
MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policies (2019, unknown): “End-to-end methods have also been proposed for training hierarchical policies [2, 7, 23, 44].”
Learning Robot Skills with Temporal Variational Inference (2020, methodology): “Our choice of VI over the forward-backward style algorithm employed in Fox et al. (2017); Krishnan et al. (2017) is because VI is amenable to both continuous and discrete latent variables z.”
DREAM Architecture: a Developmental Approach to Open-Ended Learning in Robotics (2020, unknown): “Abstract policies can also be extracted from a set of demonstrations [120] or at least ”informative” policies that may result from a former training session [121, 122].”
Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks (2021, background): “…policy through goal generation or an “options” framework [6, 8, 59], learning primitives through intrinsic motivation or other auxiliary objectives [60, 61, 62], from demonstration behavior [63, 64, 65, 66, 67], or from low-level, potentially goal conditioned, reward functions [7, 58, 68, 69].”
Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generation (2019, background): “One approach to handle long horizon tasks is to add compositional structure to policies, either from demonstrations (Krishnan et al., 2017; Fox et al., 2018), with manually-specified primitives (Xu et al.”
Modeling Long-horizon Tasks as Sequential Interaction Landscapes (2020, background): “Another direction is to learn policies for robot-object interaction from demonstrations with the goal to reduce uncertainty with expert demonstrations [64, 30, 44, 43, 42].”