Learning Rope Manipulation Policies Using Dense Object Descriptors Trained on Synthetic Depth Data

By Priya Sundaresan, Jennifer Grannen, Brijen Thananjeyan, Ashwin Balakrishna, Michael Laskey, Kevin Stone, Joseph E. Gonzalez, Ken Goldberg. 2020

145 citations recorded by the source corpus.

Robotic manipulation of deformable 1D objects such as ropes, cables, and hoses is challenging due to the lack of high-fidelity analytic models and large configuration spaces. Furthermore, learning end-to-end manipulation policies directly from images and physical interaction requires significant time on a robot and can fail to generalize across tasks. We address these challenges using interpretable deep visual representations for rope, extending recent work on dense object descriptors for robot manipulation. This facilitates the design of interpretable and transferable geometric policies built on top of the learned representations, decoupling visual reasoning and control. We present an approach that learns point-pair correspondences between initial and goal rope configurations, which implicitly encodes geometric structure, entirely in simulation from synthetic depth images. We demonstrate that the learned representation — dense depth object descriptors (DDODs) — can be used to manipulate a real rope into a variety of different arrangements either by learning from demonstrations or using interpretable geometric policies. In 50 trials of a knot-tying task with the ABB YuMi Robot, the system achieves a 66% knot-tying success rate from previously unseen configurations. See https://tinyurl.com/rope-learning for supplementary material and videos.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

Transporter Networks: Rearranging the Visual World for Robotic Manipulation (2020, background): “Alternative representations including keypoints [3, 4, 5, 6] and dense descriptors [9, 10, 11] have shown to be capable of category-level generalization and representing deformable objects, but still struggle to represent scenes with an unknown number of objects (e.”
CLIPort: What and Where Pathways for Robotic Manipulation (2021, background): “Alternatively, dense descriptors [27, 28, 29] and keypoint representations [30, 31, 32] forgo segmentation and pose representations, but do not reason about sequential actions and struggle to represent scenes with variable numbers of objects.”
Modeling, learning, perception, and control methods for deformable object manipulation (2021, methodology): “Synthetic data has been used to learn collaborative manipulation of a cloth (42) and tying a rope (115).”
Self-Supervised Goal-Conditioned Pick and Place (2020, background): “Another approach is to move objects around in order to learn about their visual appearance [3, 9, 11].”
Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Transporter Networks (2020, background): “In recent years, learning-based approaches for manipulation have grown in popularity to improve the generalization of knot-tying and also to manipulate rope towards general target configurations [52], [46], [49], [57], [64], [61].”
Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation (2021, background): “Keypoints can either be predicted directly [12, 13, 20], requiring a large, humanannotated dataset, or can be chosen among a set of selfsupervised category-level object correspondences [10, 41].”