Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation

By Anthony Simeonov, Yilun Du, Andrea Tagliasacchi, Josh Tenenbaum, Alberto Rodriguez, Pulkit Agrawal, V. Sitzmann. IEEE International Conference on Robotics and Automation, 2021

315 citations recorded by the source corpus.

We present Neural Descriptor Fields (NDFs), an object representation that encodes both points and relative poses between an object and a target (such as a robot gripper or a rack used for hanging) via category-level descriptors. We employ this representation for object manipulation, where given a task demonstration, we want to repeat the same task on a new object instance from the same category. We propose to achieve this objective by searching (via optimization) for the pose whose descriptor matches that observed in the demonstration. NDFs are conveniently trained in a self-supervised fashion via a 3D auto-encoding task that does not rely on expert-labeled keypoints. Further, NDFs are SE(3)-equivariant, guaranteeing performance that generalizes across all possible 3D object translations and rotations. We demonstrate learning of manipulation tasks from few (∼5-10) demonstrations both in simulation and on a real robot. Our performance generalizes across both object instances and 6-DoF object poses, and significantly outperforms a recent baseline that relies on 2D descriptors. Project website: https://yilundu.github.io/ndf/

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/1111653b0ec18bc18bd53f30f9ce8176e3382fd1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation (2022, background): “While other works in 6-DoF grasping [30, 31, 32, 33, 34] have used RGB-D and pointcloud input, they have not been applied to sequential tasks or used with language-conditioning.”
Foundation models in robotics: Applications, challenges, and the future (2023, unknown): “Neural descriptor ﬁelds (NDFs) [179] remove the need for dense annotation by leveraging layer-wise activations from an occupancy network; however, this approach still requires many training shapes for each target object category.”
Nerfstudio: A Modular Framework for Neural Radiance Field Development (2023, methodology): “Roboticists have explored using NeRFs for manipulation, motion planning, simulation, and mapping [Adamkiewicz et al. 2021; Byravan et al. 2022; Driess et al. 2022; Kerr et al. 2022; Simeonov et al. 2022; Zhu et al. 2022].”
Decomposing NeRF for Editing via Feature Field Distillation (2022, background): “Furthermore, while this work focuses on editable view synthesis, it is also intriguing to transfer DFF to other applications, including 3D registration of text queries [16, 1, 47, 4] or robot teaching [32, 80].”
ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation (2024, background): “On the other hand, data-driven approaches enable learning constraints directly in visual space [2, 3].”