Towards learning hierarchical skills for multi-phase manipulation tasks

By Oliver Kroemer, Christian Daniel, Gerhard Neumann, Herke Van Hoof, Jan Peters. 2015

136 citations recorded by the source corpus.

Most manipulation tasks can be decomposed into a sequence of phases, where the robot's actions have different effects in each phase. The robot can perform actions to transition between phases and, thus, alter the effects of its actions, e.g. grasp an object in order to then lift it. The robot can thus reach a phase that affords the desired manipulation. In this paper, we present an approach for exploiting the phase structure of tasks in order to learn manipulation skills more efficiently. Starting with human demonstrations, the robot learns a probabilistic model of the phases and the phase transitions. The robot then employs model-based reinforcement learning to create a library of motor primitives for transitioning between phases. The learned motor primitives generalize to new situations and tasks. Given this library, the robot uses a value function approach to learn a high-level policy for sequencing the motor primitives. The proposed method was successfully evaluated on a real robot performing a bimanual grasping task.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Control and Systems Engineering

Citations in the corpus

Learning World Graphs to Accelerate Hierarchical Reinforcement Learning (2019, neutral): “d G wis more favorable over V rand. 6 Related Works Pivotal state discovery is related to unsupervised sequence segmentation [17, 47, 76, 13, 16] and option or sub-task discovery in the context of RL [48, 6, 72, 30, 56, 53, 52]. Among them, both [52] and [76] employ sequential variational models to ”
Learning Multi-Arm Manipulation Through Collaborative Teleoperation (2020, methodology): “Reinforcement Learning (RL) has been applied to bimanual manipulation tasks [3, 7, 8, 14, 18], but RL methods must deal with the increased burden of exploration due to the presence of two arms.”
An Algorithmic Perspective on Imitation Learning (2018, methodology): “10: Learning a hierarchical skill in [Kroemer et al., 2015].”
Learning dexterous manipulation for a soft robotic hand from human demonstrations (2016, methodology): “This has been used to learn striking movements [13] and bi-manual transportation tasks [14].”
Intrinsic Motivation for Encouraging Synergistic Behavior (2020, background): “On the other hand, learning-based strategies for this task often leverage human demonstrations to circumvent the challenge of exploration (Zollner et al., 2004; Gribovskaya & Billard, 2008; Kroemer et al., 2015).”
Compositional Imitation Learning: Explaining and executing one task at a time (2018, methodology): “Our framework is closely related to option discovery [16, 17, 18, 19], with the main difference being that our inference algorithm is agnostic to what type of option (sub-task) encoding is used.”
Efficient Bimanual Manipulation Using Learned Task Schemas (2019, unknown): “ntrol settings [25], and often rely on hybrid force-position control strategies to guide both manipulators [26], [27]. These tasks have also been addressed via learning from demonstration [28], [29], [30]. In our work, we do not rely on demonstrations, and we are able to learn control policies direc”
CompILE: Compositional Imitation Learning and Execution (2018, methodology): “Our framework is closely related to option discovery (Niekum et al., 2013; Kroemer et al., 2015; Fox et al., 2017; Hausman et al., 2017; Krishnan et al., 2017; Fox et al., 2018), with the main difference being that our inference algorithm is agnostic to what type of option (sub-task) encoding is…”
A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms (2019, methodology): “Probabilistic models are thus used to capture the process noise and approximate the distributions over next states (Babaeizadeh et al., 2018; Kroemer et al., 2015).”
Modeling Long-horizon Tasks as Sequential Interaction Landscapes (2020, methodology): “physics-based heuristics [2, 55], hierarchical planning [24, 36], probabilistic models [31, 17] or integrating action parameters as part of learning forward models [20].”
Multi-Modal Imitation Learning from Unstructured Demonstrations using Generative Adversarial Nets (2017, unknown): “Examples include [23, 18, 14, 31, 13].”
Variable Impedance Control in End-Effector Space: An Action Space for Reinforcement Learning in Contact-Rich Tasks (2019, background): “), the generation of reference signals by g(·) can be addressed in multiple ways, ranging from hand-tuned state machines [15, 41], trajectory optimization [14, 45], imitation learning [5, 12, 19, 38], or reinforcement learning (RL) [9, 23].”
Discovering Motor Programs by Recomposing Demonstrations (2020, background): “Note that this notion of ‘plannability’ as ‘simplicity’ is merely one plausible alternative, and alternate ones can be explored e.g. ‘linearity’ (Kroemer et al., 2015) or ‘predictability of motion’ (Wolpert & Kawato, 1998).”
Stable reinforcement learning with autoencoders for tactile and visual data (2016, background): “A popular policy parametrization uses dynamic motor primitives to generate trajectories to be tracked [4], [5], [11]–[14].”
Continuous Control with Action Quantization from Demonstrations (2021, background): “…explored the learning of primitives or options from demonstrations together with a high-level controller that is either learned from demonstrations (Kroemer et al., 2015; Krishnan et al., 2017; Le et al., 2018; Ding et al., 2019; Lynch et al., 2020), or learned from interactions with the…”
Recent Advances in Robot Learning from Demonstration (2020, background): “(148) utilized demonstrations to learn a stochastic model of the phases and transitions involved in manipulation skills and then used reinforcement learning to learn how to transition between the previously identified learned phases.”
KeyIn: Discovering Subgoal Structure with Keyframe-based Video Prediction (2019, background): “L G ] 11 A p r 2019 once a subgoal sequence is discovered (Konidaris et al., 2012; Niekum et al., 2015; Kroemer et al., 2015; Hausman et al., 2017).”