Manipulation by Feel: Touch-Based Control with Deep Predictive Models

By Stephen Tian, Frederik D. Ebert, Dinesh Jayaraman, Mayur Mudigonda, Chelsea Finn, Roberto Calandra, Sergey Levine. 2019

154 citations recorded by the source corpus.

Touch sensing is widely acknowledged to be important for dexterous robotic manipulation, but exploiting tactile sensing for continuous, non-prehensile manipulation is challenging. General purpose control techniques that are able to effectively leverage tactile sensing as well as accurate physics models of contacts and forces remain largely elusive, and it is unclear how to even specify a desired behavior in terms of tactile percepts. In this paper, we take a step towards addressing these issues by combining high-resolution tactile sensing with data-driven modeling using deep neural network dynamics models. We propose deep tactile MPC, a framework for learning to perform tactile servoing from raw tactile sensor inputs, without manual supervision. We show that this method enables a robot equipped with a GelSight-style tactile sensor to manipulate a ball, analog stick, and 20-sided die, learning from unsupervised autonomous interaction and then using the learned tactile predictive model to reposition each object to user-specified configurations, indicated by a goal tactile reading. Videos, visualizations and the code are available here: https://sites.google.com/view/deeptactilempc

Subfields: Biomedical Engineering, Cognitive Neuroscience

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Cognitive Neuroscience

Citations in the corpus

RoboNet: Large-Scale Multi-Robot Learning (2019, methodology): “This method is related to a range of recently proposed techniques that use predictive models for vision-based control [28, 29, 30, 31, 32].”
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor With Application to In-Hand Manipulation (2020, methodology): “In comparison, CDNA would take 69 seconds for a single step, making it impractical to use for control.”
A Review of Tactile Information: Perception and Action Through Touch (2020, background): “Another nonprehensile tactile controller was learned [181] for rolling tasks.”
Cable manipulation with a tactile-reactive gripper (2019, methodology): “[23] proposed a tactile-based model predictive control method to reposition an object.”
Improvisation through Physical Understanding: Using Novel Objects as Tools with Visual Foresight (2019, background): “work has shown that approaches like visual foresight can be integrated with tactile sensor inputs [42]; but in practice, the introduction of tactile or force sensors would likely introduce additional challenges in evaluating predictions and collecting data safely.”