3D Shape Perception from Monocular Vision, Touch, and Shape Priors

By Shaoxiong Wang, Jiajun Wu, Xingyuan Sun, Wenzhen Yuan, William T. Freeman, Josh Tenenbaum, Edward H. Adelson. 2018

133 citations recorded by the source corpus.

Perceiving accurate 3D object shape is important for robots to interact with the physical world. Current research along this direction has been primarily relying on visual observations. Vision, however useful, has inherent limitations due to occlusions and the 2D-3D ambiguities, especially for perception with a monocular camera. In contrast, touch gets precise local shape information, though its efficiency for reconstructing the entire shape could be low. In this paper, we propose a novel paradigm that efficiently perceives accurate 3D object shape by incorporating visual and tactile observations, as well as prior knowledge of common object shapes learned from large-scale shape repositories. We use vision first, applying neural networks with learned shape priors to predict an object's 3D shape from a single-view color image. We then use tactile sensing to refine the shape; the robot actively touches the object regions where the visual prediction has high uncertainty. Our method efficiently builds the 3D shape of common objects from a color image and a small number of tactile explorations (around 10). Our setup is easy to apply and has potentials to help robots better perform grasping or manipulation tasks on real-world objects.

Subfields: Computational Mechanics, Geology, Industrial and Manufacturing Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Geology

In Industrial and Manufacturing Engineering

Citations in the corpus

Learning Shape Priors for Single-View 3D Completion and Reconstruction (2018, background): “While most of these approaches encode objects in voxels from vision, there have also been attempts to reconstruct objects in point clouds [11,15] or octave trees [40,49,39], or using tactile signals [53].”
Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Tasks (2019, background): “Several works in perception and state estimation have combined visual and tactile information for object tracking [46]–[48], shape completion [49], and scene understanding [50], [51].”
Vision-based robotic grasping from object localization, object pose estimation to grasp estimation for parallel grippers: a review (2020, methodology): “2020), and some other methods (Wang et al. 2018; Yan et al. 2018, 2019; Gao and Tedrake 2019; Sajjan et al. 2019) utilize the RGB-D images.”
A Review of Tactile Information: Perception and Action Through Touch (2020, background): “Employing tactile sensing is especially useful for estimating the shape of an object in visually occluded regions [125]–[127].”