The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?

By Roberto Calandra, Andrew Owens, M. Upadhyaya, Wenzhen Yuan, Justin Lin, Edward H. Adelson, Sergey Levine. 2017

247 citations recorded by the source corpus.

A successful grasp requires careful balancing of the contact forces. Deducing whether a particular grasp will be successful from indirect measurements, such as vision, is therefore quite challenging, and direct sensing of contacts through touch sensing provides an appealing avenue toward more successful and consistent robotic grasping. However, in order to fully evaluate the value of touch sensing for grasp outcome prediction, we must understand how touch sensing can influence outcome prediction accuracy when combined with other modalities. Doing so using conventional model-based techniques is exceptionally difficult. In this work, we investigate the question of whether touch sensing aids in predicting grasp outcomes within a multimodal sensing framework that combines vision and touch. To that end, we collected more than 9,000 grasping trials using a two-finger gripper equipped with GelSight high-resolution tactile sensors on each finger, and evaluated visuo-tactile deep neural network models to directly predict grasp outcomes from either modality individually, and from both modalities together. Our experimental results indicate that incorporating tactile readings substantially improve grasping performance.

Subfields: Biomedical Engineering, Cognitive Neuroscience, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Cognitive Neuroscience

In Control and Systems Engineering

Citations in the corpus

Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control (2018, background): “A number of recent works have studied self-supervised robotic learning, where large-scale unattended data collection is used to learn individual skills such as grasping [22], [23], [24], [25], push-grasp synergies [26], or obstacle avoidance [27], [28].”
Learning Awareness Models (2018, background): “There is also work in robotics on using the anticipation of sensation to guide actions (Indranil Sur, 2017), and on showing how touch sensing can improve the performance of grasping tasks (Calandra et al., 2017).”
Robot Learning in Homes: Improving Generalization and Reducing Dataset Bias (2018, background): “Erickson et al. [28], Murali et al. [29], and Calandra et al. [30] collected a dataset of robotic tactile interactions for material recognition and grasp stability estimation.”
Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks (2018, background): “Many of them are concerned with estimating the stability of a grasp before lifting an object [6, 14], even suggesting a regrasp [50].”
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor With Application to In-Hand Manipulation (2020, background): “Touch is a crucial sensory modality for both humans [1] and robots [2], as it”
Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Tasks (2019, background): “Many of them are concerned with estimating the stability of a grasp before lifting an object [15], [16], even suggesting a regrasp [17].”
Manipulation by Feel: Touch-Based Control with Deep Predictive Models (2019, background): “versatile, high-bandwidth, high-spatial resolution alternative to traditional tactile sensing [7], [8].”
Learning robust, real-time, reactive robotic grasping (2019, background): “Integrating other sensors such as touch (Calandra et al., 2017) or allowing the robot to execute other manipulation actions such as pushing (Zeng et al., 2018a) are also both powerful ways to improve the robustness of a closed-loop robotic grasping system in which the robot is an active and…”
ReSkin: versatile, replaceable, lasting tactile skins (2021, background): “Machine learning approaches have recently been shown to effectively parse soft sensor data for robotic grasping [7, 8, 9], proprioception [10], and object classification [11], among many others [12, 13, 14].”
GelSlim: A High-Resolution, Compact, Robust, and Calibrated Tactile-sensing Finger (2018, methodology): “GelSight sensors have also been used to estimate grasp quality [18].”
A Review of Tactile Information: Perception and Action Through Touch (2020, unknown): “and object-level [115], [123], [134], [135] information as well as directly from tactile signals [27], [144]–[147].”
More Than a Feeling: Learning to Grasp and Regrasp Using Vision and Touch (2018, background): “future grasp success for novel objects? Recall that while previous works such as [18] have shown that it is possible to predict stability of ongoing grasps from visuo-tactile inputs, we seek to”