KeyPose: Multi-View 3D Labeling and Keypoint Estimation for Transparent Objects
By Xingyu Liu, Rico Jonschkowski, Anelia Angelova, Kurt Konolige. 2019
145 citations recorded by the source corpus.
Estimating the 3D pose of desktop objects is crucial for applications such as robotic manipulation. Many existing approaches to this problem require a depth map of the object for both training and prediction, which restricts them to opaque, lambertian objects that produce good returns in an RGBD sensor. In this paper we forgo using a depth sensor in favor of raw stereo input. We address two problems: first, we establish an easy method for capturing and labeling 3D keypoints on desktop objects with an RGB camera; and second, we develop a deep neural network, called KeyPose, that learns to accurately predict object poses using 3D keypoints, from stereo input, and works even for transparent objects. To evaluate the performance of our method, we create a dataset of 15 clear objects in five classes, with 48K 3D-keypoint labeled images. We train both instance and category models, and show generalization to new textures, poses, and objects. KeyPose surpasses state-of-the-art performance in 3D pose estimation on this dataset by factors of 1.5 to 3.5, even in cases where the competing method is provided with ground-truth depth. Stereo input is essential for this performance as it improves results compared to using monocular input by a factor of 2. We will release a public version of the data capture and labeling pipeline, the transparent object database, and the KeyPose models and evaluation code. Project website: https://sites.google.com/corp/view/keypose.
Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Control and Systems Engineering
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Aerospace Engineering
- Lucas Manuelli
- Peter R. Florence
- Dieter Fox
- Andy Zeng
- Mohit Shridhar
- Johnny Lee
- Jonathan Tompson
- Shuran Song
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dieter Fox
- Li Fei-Fei
- Jitendra Malik
- Trevor Darrell
- Andy Zeng
- Cordelia Schmid
- Andrew Zisserman
In Control and Systems Engineering
- Dieter Fox
- Lucas Manuelli
- Mohit Shridhar
- Peter R. Florence
- Andy Zeng
- Johnny Lee
- Jonathan Tompson
- Vikas Sindhwani
Citations in the corpus
- Transporter Networks: Rearranging the Visual World for Robotic Manipulation (2020, background): “Alternative representations including keypoints [3, 4, 5, 6] and dense descriptors [9, 10, 11] have shown to be capable of category-level generalization and representing deformable objects, but still struggle to represent scenes with an unknown number of objects (e.”
- CLIPort: What and Where Pathways for Robotic Manipulation (2021, background): “Alternatively, dense descriptors [27, 28, 29] and keypoint representations [30, 31, 32] forgo segmentation and pose representations, but do not reason about sequential actions and struggle to represent scenes with variable numbers of objects.”