A Practical Approach to Insertion with Variable Socket Position Using Deep Reinforcement Learning

By Mel Vecerík, Oleg O. Sushkov, David Barker, Thomas Rothörl, Todd Hester, Jonathan Scholz. 2018

118 citations recorded by the source corpus.

Insertion is a challenging haptic and visual control problem with significant practical value for manufacturing. Existing approaches in the model-based robotics community can be highly effective when task geometry is known, but are complex and cumbersome to implement, and must be tailored to each individual problem by a qualified engineer. Within the learning community there is a long history of insertion research, but existing approaches are either too sample-inefficient to run on real robots, or assume access to high-level object features, e.g. socket pose. In this paper we show that relatively minor modifications to an off-the-shelf Deep-RL algorithm (DDPG), combined with a small number of human demonstrations, allows the robot to quickly learn to solve these tasks efficiently and robustly. Our approach requires no modeling or simulation, no parameterized search or alignment behaviors, no vision system aside from raw images, and no reward shaping. We evaluate our approach on a narrow-clearance peg-insertion task and a deformable clip-insertion task, both of which include variability in the socket position. Our results show that these tasks can be solved reliably on the real robot in less than 10 minutes of interaction time, and that the resulting policies are robust to variance in the socket position and orientation.

Subfields: Aerospace Engineering, Artificial Intelligence, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Artificial Intelligence

In Control and Systems Engineering

Citations in the corpus

A Framework for Data-Driven Robotics (2019, methodology): “Binary success labels of frames are also used to learn reward functions [26], [60].”
End-to-End Robotic Reinforcement Learning without Reward Engineering (2019, methodology): “Prior work generally requires both successful and failed examples to be part of D [42, 40].”
Challenges of real-world reinforcement learning: definitions, benchmarks and analysis (2021, methodology): “We are inspired by a large range of real-world tasks, from control systems grounded in the physical world (Vecerik et al. 2019; Kalashnikov et al. 2018) to global-scale software systems interacting with billions of users (Gauci et al. 2018; Covington et al. 2016; Ie et al. 2019).”
XIRL: Cross-embodiment Inverse Reinforcement Learning (2021, methodology): “Note that unlike [27], we do not have 490 access to any unsuccessful trajectories in the dataset.”
Recent Advances in Robot Learning from Demonstration (2020, background): “In an effort to leverage the benefits of LfD’s data efficiency, researchers have demonstrated that LfD can be combined with reinforcement learning in order to improve sample efficiency (189, 190, 191, 192, 193, 194, 195, 196).”
An empirical investigation of the challenges of real-world reinforcement learning (2020, methodology): “This approach has been combined with DQN [Mnih et al. 2015] and demonstrated on Atari [Hester et al. 2018a], as well as combined with DDPG [Lillicrap et al. 2015] for insertion tasks on robots [Vecerı́k et al. 2019].”
Scaling data-driven robotics with reward sketching and batch reinforcement learning (2019, methodology): “As in prior works [67, 54, 68], we use demonstrations as part of the agent experience and train with temporal difference learning in a model-free setting.”
Learning Deep Parameterized Skills from Demonstration for Re-targetable Visuomotor Control (2019, methodology): “e efﬁcient, is end-to-end, and can be instructed to speciﬁc goals. There has also been work in imitation learning that takes raw sensory data from human demonstrations and learns robot motor controls [2, 3, 10, 18, 19, 23, 25, 26]. Our model speciﬁcally takes inspiration from the behavioral cloning ”
Positive-Unlabeled Reward Learning (2019, methodology): “Possible forms of annotations include human rating or ranking of states or trajectories (Ibarz et al., 2018; Vecerik et al., 2019; Cabi et al., 2019).”