Reinforcement learning for non-prehensile manipulation: Transfer from simulation to physical system

By Kendall Lowrey, Svetoslav Kolev, Jeremy Dao, Aravind Rajeswaran, E. Todorov. 2018

66 citations recorded by the source corpus.

Reinforcement learning has emerged as a promising methodology for training robot controllers. However, most results have been limited to simulation due to the need for a large number of samples and the lack of automated-yet-safe data collection methods. Model-based reinforcement learning methods provide an avenue to circumvent these challenges, but the traditional concern has been the mismatch between the simulator and the real world. Here, we show that control policies learned in simulation can successfully transfer to a physical system, composed of three Phantom robots pushing an object to various desired target positions. We use a modified form of the natural policy gradient algorithm for learning, applied to a carefully identified simulation model. The resulting policies, trained entirely in simulation, work well on the physical system without additional training. In addition, we show that training with an ensemble of models makes the learned policies more robust to modeling errors, thus compensating for difficulties in system identification. The results are illustrated in the accompanying video.

Subfields: Artificial Intelligence, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

Citations in the corpus

Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience (2018, background): “Combination of system identification and dynamics randomization has been used in the past to learn locomotion for a real quadruped [26], non-prehensile object manipulation [27] and in-hand object pivoting [28].”
Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control (2018, background): “Nominal dynamics models based on knowledge of physics [1], or through learning [2], complements a growing body of work on successful simulation to reality transfer and system identification [3, 4, 5, 6, 7].”
TriFinger: An Open-Source Robot for Learning Dexterity (2020, methodology): “The two setups which are most similar to the TriFinger are the D’Claw, a three-ﬁngered robotic hand (Ahn et al., 2019) and the Phantom Manipulation Platform, consisting of three Phantom Haptic Devices (Lowrey et al., 2018), see ﬁgure 2.”
Learning Agile Robotic Locomotion Skills by Imitating Animals (2020, background): “Sim-toreal transfer can be facilitated by constructing more accurate simulations [58, 62], or adapting the simulator with realworld data [57, 23, 26, 36, 5].”