Learning to Control a Low-Cost Manipulator using Data-Efficient Reinforcement Learning

By Marc Peter Deisenroth, Carl E. Rasmussen, Dieter Fox. 2011

250 citations recorded by the source corpus.

Over the last years, there has been substantial progress in robust manipulation in unstructured environments. The long-term goal of our work is to get away from precise, but very expensive robotic systems and to develop affordable, potentially imprecise, self-adaptive manipulator systems that can interactively perform tasks such as playing with children. In this paper, we demonstrate how a low-cost off-the-shelf robotic system can learn closed-loop policies for a stacking task in only a handful of trials-from scratch. Our manipulator is inaccurate and provides no pose feedback. For learning a controller in the work space of a Kinect-style depth camera, we use a model-based reinforcement learning technique. Our learning method is data efficient, reduces model bias, and deals with several noise sources in a principled way during long-term planning. We present a way of incorporating state-space constraints into the learning process and analyze the learning gain by exploiting the sequential structure of the stacking task.

Subfields: Artificial Intelligence, Biomedical Engineering, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

End-to-End Training of Deep Visuomotor Policies (2015, background): “They have been applied to a range of robotic tasks, including manipulation [2, 13] and locomotion [5, 7, 15, 39].”
Reinforcement learning in robotics: A survey (2013, methodology): “By treating model uncertainty as if it were noise (Schneider, 1996) as well as employing analytic approximations of forward simulation, a cart-pole task can be solved with less than 20 seconds of interaction with the physical system (Deisenroth and Rasmussen, 2011); a visually driven block-stacking ”
Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning (2017, background): “Model-based algorithms are known in general to outperform model-free learners in terms of sample complexity [4], and in practice have been applied successfully to control both simulated and real-world robotic systems, such as pendulums [6], legged robots [18], swimmers [19], and manipulators [20].”
Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience (2018, methodology): “In [13, 14], the data from real world policy executions is used to fit a probabilistic dynamics model, which is then used for learning an optimal policy.”
Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees (2018, methodology): “…and has been successfully applied to robotics in both simulation and in the real world (Deisenroth & Rasmussen, 2011b; Morimoto & Atkeson, 2003; Deisenroth et al., 2011) using dynamical models ranging from Gaussian process (Deisenroth & Rasmussen, 2011b; Ko & Fox, 2009), time-varying linear…”
Deep spatial autoencoders for visuomotor learning (2015, methodology): “Reinforcement learning and policy search methods have been applied to perform a variety of robotic tasks, such as table tennis [9], object manipulation [10], [11], and locomotion [12], [13], [14], [15].”
Overcoming Exploration in Reinforcement Learning with Demonstrations (2017, methodology): “Stacking blocks has been studied before in the literature [3], [4] and exhibits many of the difficulties mentioned: long horizons, contacts, and requires generalizing to each instance of the task.”
Path integral guided policy search (2016, methodology): “Reinforcement learning (RL) and policy search methods have shown considerable promise for enabling robots to automatically learn a wide range of complex skills [1, 2, 3, 4], and recent results in deep reinforcement learning suggest that this capability can be extended to learn nonlinear policies tha”
Learning contact-rich manipulation skills with guided policy search (2015, background): “table tennis [11], manipulation [9], [12], and robotic locomo-”
Learning robot in-hand manipulation with tactile features (2015, methodology): “These methods are driven by feedback from tactile sensing [12, 35], Cartesian- and joint-space coordinates [33, 34], or both [32].”
Robot Learning in Homes: Improving Generalization and Reducing Dataset Bias (2018, methodology): “[35] used model-based RL to teach a cheap inaccurate 6 DOF robot to stack multiple blocks.”
Multi-task policy search for robotics (2014, methodology): “The plastic arm could be controlled by commanding both a desired conﬁguration of the six servos and the duration for executing the command [11].”
PILCO: A Model-Based and Data-Efficient Approach to Policy Search (2011, unknown): “An extension of pilco to deal with planning and control under consideration of task-space constraints in a robotic manipulation task is presented in (Deisenroth et al., 2011).”
Learning Visual Feature Spaces for Robotic Manipulation with Deep Spatial Autoencoders (2015, methodology): “Reinforcement learning and policy search methods have been applied to perform a variety of robotic tasks, such as table tennis [9], object manipulation [10], [11], and locomotion [12], [13], [14], [15].”
Fast Model Identification via Physics Engines for Data-Efficient Policy Search (2017, methodology): “The method is compared to PoWER [ Kober and Peters, 2009 ] and PILCO [Deisenroth et al. , 2011].”
Manipulators and Manipulation in high dimensional spaces (2016, background): “Indeed RL has been successfully applied to a range of robotic tasks [26, 43, 78, 101], however dimensionality and sample complexity have presented major challenges [25, 42].”
Model-Based Reinforcement Learning for Closed-Loop Dynamic Control of Soft Robotic Manipulators (2019, methodology): “In [16], an algorithm called probabilistic inference for learn-”