More Than a Feeling: Learning to Grasp and Regrasp Using Vision and Touch

By Roberto Calandra, Andrew Owens, Dinesh Jayaraman, Justin Lin, Wenzhen Yuan, Jitendra Malik, Edward H. Adelson, Sergey Levine. IEEE Robotics and Automation Letters, 2018

429 citations recorded by the source corpus.

For humans, the process of grasping an object relies heavily on rich tactile feedback. Most recent robotic grasping work, however, has been based only on visual input, and thus cannot easily benefit from feedback after initiating contact. In this letter, we investigate how a robot can learn to use tactile information to iteratively and efficiently adjust its grasp. To this end, we propose an end-to-end action-conditional model that learns regrasping policies from raw visuo-tactile data. This model—a deep, multimodal convolutional network—predicts the outcome of a candidate grasp adjustment, and then executes a grasp by iteratively selecting the most promising actions. Our approach requires neither calibration of the tactile sensors nor any analytical modeling of contact forces, thus reducing the engineering effort required to obtain efficient grasping policies. We train our model with data from about 6450 grasping trials on a two-finger gripper equipped with GelSight high-resolution tactile sensors on each finger. Across extensive experiments, our approach outperforms a variety of baselines at 1) estimating grasp adjustment outcomes, 2) selecting efficient grasp adjustments for quick grasping, and 3) reducing the amount of force applied at the fingers, while maintaining competitive performance. Finally, we study the choices made by our model and show that it has successfully acquired useful and interpretable grasping behaviors.

Subfields: Biomedical Engineering, Cognitive Neuroscience, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/4ff5b0b368f06cb1efb8d62e6490c43cd746ae40

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Cognitive Neuroscience

In Control and Systems Engineering

Citations in the corpus

Learning ambidextrous robot grasping policies (2019, background): “Another way to increase rate is to use feedback policies that actively regrasp dropped objects based on visual servoing ( 10 , 28 , 42 ), force sensing ( 43 – 45 ), or tactile sensing ( 46 – 49 ).”
Manipulation by Feel: Touch-Based Control with Deep Predictive Models (2019, methodology): “GelSight-style sensors have been used in a number of prior works for other applications, including tracking [25], inserting USB connectors [26], estimating object hardness [27], and grasping [28].”
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor With Application to In-Hand Manipulation (2020, methodology): “for in-hand manipulation, accomplishing a rolling task of a large object between two fingers of a robotic hand using MEMS barometers as tactile feedback [28] The GelSight sensor in particular has also shown success in learning models to predict grasp success of complex and varying geometries [29].”
Cable manipulation with a tactile-reactive gripper (2019, methodology): “tion from the GelSight sensor. Izatt et al. [19] used the 3D point cloud from GelSight sensor in a Kalman ﬁlter to better register the position of a screwdriver in a peg-in-hole task. Calandra et al. [20] and Hogan et al. [21] demonstrated that the tactile images from GelSight and GelSlim sensors co”
A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms (2019, background): “, a robot can use tactile sensing to determine if a grasp attempt succeeded Grasped(obj)=True [52, 75, 254, 76].”
A Review of Tactile Information: Perception and Action Through Touch (2020, neutral): “driven approaches [150], [167], [201].”
Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks (2018, background): “Some of them extend their previous work on grasp stability estimation [5, 13].”
Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Tasks (2019, background): “Some of them extend their previous work on grasp stability estimation [24], [25] and shape completion for grasping [26].”
ReSkin: versatile, replaceable, lasting tactile skins (2021, background): “Machine learning approaches have recently been shown to effectively parse soft sensor data for robotic grasping [7, 8, 9], proprioception [10], and object classification [11], among many others [12, 13, 14].”