Self-Supervised Learning of State Estimation for Manipulating Deformable Linear Objects

By Mengyuan Yan, Yilin Zhu, Ning Jin, Jeannette Bohg. 2019

184 citations recorded by the source corpus.

We demonstrate model-based, visual robot manipulation of deformable linear objects. Our approach is based on a state-space representation of the physical system that the robot aims to control. This choice has multiple advantages, including the ease of incorporating physics priors in the dynamics model and perception model, and the ease of planning manipulation actions. In addition, physical states can naturally represent object instances of different appearances. Therefore, dynamics in the state space can be learned in one setting and directly used in other visually different settings. This is in contrast to dynamics learned in pixel space or latent space, where generalization to visual differences are not guaranteed. Challenges in taking the state-space approach are the estimation of the high-dimensional state of a deformable object from raw images, where annotations are very expensive on real data, and finding a dynamics model that is both accurate, generalizable, and efficient to compute. We are the first to demonstrate self-supervised training of rope state estimation on real images, without requiring expensive annotations. This is achieved by our novel self-supervising learning objective, which is generalizable across a wide range of visual appearances. With estimated rope states, we train a fast and differentiable neural network dynamics model that encodes the physics of mass-spring systems. Our method has a higher accuracy in predicting future states compared to models that do not involve explicit state estimation and do not use any physics prior, while only using 3% of training data. We also show that our approach achieves more efficient manipulation, both in simulation and on a real robot, when used within a model predictive controller.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation (2020, background): “Recently, the success of deep learning has garnered increased interest in learning to solve perception and manipulation tasks in an end-to-end framework [11, 12, 28, 29, 30].”
Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Transporter Networks (2020, background): “In recent years, learning-based approaches for manipulation have grown in popularity to improve the generalization of knot-tying and also to manipulate rope towards general target configurations [52], [46], [49], [57], [64], [61].”
Challenges and Outlook in Robotic Manipulation of Deformable Objects (2021, methodology): “Vision is used in tasks such as rope manipulation [25], [26] or cloth unfolding [27], [28], where the object exhibits large global deformation.”
Cable manipulation with a tactile-reactive gripper (2019, background): “For this reason, much work on cables (and other deformable linear objects) has utilized mechanical constraints [1, 2, 3].”