Learning compound multi-step controllers under unknown dynamics

By Weiqiao Han, Sergey Levine, Pieter Abbeel. 2015

54 citations recorded by the source corpus.

Applications of reinforcement learning for robotic manipulation often assume an episodic setting. However, controllers trained with reinforcement learning are often situated in the context of a more complex compound task, where multiple controllers might be invoked in sequence to accomplish a higher-level goal. Furthermore, training such controllers typically requires resetting the environment between episodes, which is typically handled manually. We describe an approach for training chains of controllers with reinforcement learning. This requires taking into account the state distributions induced by preceding controllers in the chain, as well as automatically training reset controllers that can reset the task between episodes. The initial state of each controller is determined by the controller that precedes it, resulting in a non-stationary learning problem. We demonstrate that a recently developed method that optimizes linear-Gaussian controllers under learned local linear models can tackle this sort of non-stationary problem, and that training controllers concurrently with a corresponding reset controller only minimally increases training time. We also demonstrate this method on a complex tool use task that consists of seven stages and requires using a toy wrench to screw in a bolt. This compound task requires grasping and handling complex contact dynamics. After training, the controllers can execute the entire task quickly and efficiently. Finally, we show that this method can be combined with guided policy search to automatically train nonlinear neural network controllers for a grasping task with considerable variation in target position.

Subfields: Artificial Intelligence, Biomedical Engineering, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Control and Systems Engineering

Citations in the corpus

Learning to Optimize (2016, methodology): “autonomous algorithm” or “policy”. We use an off-the-shelf reinforcement learning algorithm known as guided policy search [17], which has demonstrated success in a variety of robotic control settings [18, 10, 19, 12]. We show empirically that the autonomous optimization algorithm we learn converges ”
Continual Learning of Control Primitives: Skill Discovery via Reset-Games (2020, background): “Learning without access to resets has been studied in prior work with a focus on automation, safety, and learning compound controllers [23, 13, 8].”
Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning (2017, background): “Han et al. (2015) learn a sequence of forward and reset policies to complete a complex manipulation task.”
Model-Based Reinforcement Learning for Closed-Loop Dynamic Control of Soft Robotic Manipulators (2019, methodology): “The state-of-the-art approaches using variations of this idea are concerned with using learned local models [20], compound multistep controllers [21], and deep representation of the control policies [22].”
AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videos (2019, result): “This is similar to prior work on learning reset controllers [20], [21].”
Recovery RL: Safe Reinforcement Learning With Learned Recovery Zones (2020, background): “Han et al. [16] and Eysenbach et al. [10] introduce reset policies which are trained jointly with the task policy to reset the agent to its initial state distribution, ensuring that the task policy only learns behaviors which can be reset [10].”
Autonomous Reinforcement Learning via Subgoal Curricula (2021, background): “Note, this ˜ r loosely recovers the objectives used in some prior works [19, 6].”
The Ingredients of Real-World Robotic Reinforcement Learning (2020, background): “Learning without resets has been considered in prior works (Eysenbach et al., 2018; Han et al., 2015), although in different contexts – safe learning and learning compound controllers, respectively.”
Reset-Free Lifelong Learning with Skill-Space Planning (2020, background): “n-depth discussion of the related works is in Appendix A. Non-Episodic RL. The traditional approach to non-episodic RL is to explicitly learn a policy to reset the environment (Even-Dar et al., 2005; Han et al., 2015; Eysenbach et al., 2017). This requires somewhat hard-to-deﬁne notions of what a re”
Interactive Perception: Leveraging Action in Perception and Perception in Action (2016, unknown): “Less obvious is the result by Kjellström et al. [11] who showed that classifying objects is easier if they are observed while being used by a person.”