kPAM: KeyPoint Affordances for Category-Level Robotic Manipulation

By Lucas Manuelli, Peter R. Florence, Russ Tedrake. 2019

311 citations recorded by the source corpus.

We would like robots to achieve purposeful manipulation by placing any instance from a category of objects into a desired set of goal states. Existing manipulation pipelines typically specify the desired configuration as a target 6-DOF pose and rely on explicitly estimating the pose of the manipulated objects. However, representing an object with a parameterized transformation defined on a fixed template cannot capture large intra-category shape variation, and specifying a target pose at a category level can be physically infeasible or fail to accomplish the task -- e.g. knowing the pose and size of a coffee mug relative to some canonical mug is not sufficient to successfully hang it on a rack by its handle. Hence we propose a novel formulation of category-level manipulation that uses semantic 3D keypoints as the object representation. This keypoint representation enables a simple and interpretable specification of the manipulation target as geometric costs and constraints on the keypoints, which flexibly generalizes existing pose-based manipulation methods. Using this formulation, we factor the manipulation policy into instance segmentation, 3D keypoint detection, optimization-based robot action planning and local dense-geometry-based action execution. This factorization allows us to leverage advances in these sub-problems and combine them into a general and effective perception-to-action manipulation pipeline. Our pipeline is robust to large intra-category shape variation and topology changes as the keypoint representation ignores task-irrelevant geometric details. Extensive hardware experiments demonstrate our method can reliably accomplish tasks with never-before seen objects in a category, such as placing shoes and mugs with significant shape variation into category level target configurations.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Control and Systems Engineering, 3D Vision & Pose, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In 3D Vision & Pose

In Robotics & Control

Citations in the corpus

Generalization Through Hand-Eye Coordination: An Action Space for Learning Spatially-Invariant Visuomotor Control (2021, methodology): “Keypoints have been shown to be an effective representation for visuomotor policy learning [8, 14, 21, 22, 26].”
Transporter Networks: Rearranging the Visual World for Robotic Manipulation (2020, background): “Alternative representations including keypoints [3, 4, 5, 6] and dense descriptors [9, 10, 11] have shown to be capable of category-level generalization and representing deformable objects, but still struggle to represent scenes with an unknown number of objects (e.”
Self-Supervised Correspondence in Visuomotor Policy Learning (2019, methodology): “Future work could, similar to the visual pipeline in [40], combine both instance-level segmentation with intra-instance visual representations.”
CLIPort: What and Where Pathways for Robotic Manipulation (2021, background): “Alternatively, dense descriptors [27, 28, 29] and keypoint representations [30, 31, 32] forgo segmentation and pose representations, but do not reason about sequential actions and struggle to represent scenes with variable numbers of objects.”
ContactNets: Learning of Discontinuous Contact Dynamics with Smooth, Implicit Representations (2020, background): “Recent advances in keypoint-based approaches [35] suggest a promising intermediate representation for inferring contact geometry from video.”
Keypoints into the Future: Self-Supervised Correspondence in Model-Based Reinforcement Learning (2020, background): “And as discussed in prior works [8, 18], keypoints and dense correspondences provide advantages over using 6D object poses: they can apply to deformable objects and represent category-level generalization.”
KeyPose: Multi-View 3D Labeling and Keypoint Estimation for Transparent Objects (2019, methodology): “Previous work has explored deep learning methods for detecting 3D keypoints of an object given a monocular RGB image [26] or RGBD image [15].”
iNeRF: Inverting Neural Radiance Fields for Pose Estimation (2020, methodology): “range of applications, including robot manipulation, and mobile robotics, and augmented reality, [16], [17], [6].”
ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation (2024, methodology): “Among them, keypoints have shown great promises given their interpretability, efficiency, generalization to instance variations [4], and ability to model both rigid bodies and deformable objects.”
Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes (2021, background): “Model-based grasping circumvents reasoning about the physics of contact and grasp generation by pre-defining a set of grasps in the object frame and transform those grasps according to the 6-DoF object pose [1, 2, 3, 4] or detected keypoints of the objects [5, 6].”
Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation (2021, background): “Keypoints can either be predicted directly [12, 13, 20], requiring a large, human-annotated dataset, or can be chosen among a set of self-supervised category-level object correspondences [10, 41].”
TAP-Vid: A Benchmark for Tracking Any Point in a Video (2022, unknown): “…bring solutions to many interesting challenges, e.g., better handling of dynamic or deformable objects in SFM [56], and allowing the semantic keypoint-based methods [15, 42, 43, 72] often employed in robotic object manipulation to be applied to generic objects, independent of their class.”
KETO: Learning Keypoint Representations for Tool Manipulation (2019, methodology): “In contrast to previous work [34], the keypoint prediction is learned from self-supervised robot interactions in the task environment.”