Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation

By Peter R. Florence, Lucas Manuelli, Russ Tedrake. 2018

311 citations recorded by the source corpus.

What is the right object representation for manipulation? We would like robots to visually perceive scenes and learn an understanding of the objects in them that (i) is task-agnostic and can be used as a building block for a variety of manipulation tasks, (ii) is generally applicable to both rigid and non-rigid objects, (iii) takes advantage of the strong priors provided by 3D vision, and (iv) is entirely learned from self-supervision. This is hard to achieve with previous methods: much recent work in grasping does not extend to grasping specific objects or other tasks, whereas task-specific learning may require many trials to generalize well across object configurations or other tasks. In this paper we present Dense Object Nets, which build on recent developments in self-supervised dense descriptor learning, as a consistent object representation for visual understanding and manipulation. We demonstrate they can be trained quickly (approximately 20 minutes) for a wide variety of previously unseen and potentially non-rigid objects. We additionally present novel contributions to enable multi-object descriptor learning, and show that by modifying our training procedure, we can either acquire descriptors which generalize across classes of objects, or descriptors that are distinct for each object instance. Finally, we demonstrate the novel application of learned dense descriptors to robotic manipulation. We demonstrate grasping of specific points on an object across potentially deformed object configurations, and demonstrate using class general descriptors to transfer specific grasps across objects in a class.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Generalization Through Hand-Eye Coordination: An Action Space for Learning Spatially-Invariant Visuomotor Control (2021, background): “Keypoints have been shown to be an effective representation for visuomotor policy learning [8, 14, 21, 22, 26].”
Transporter Networks: Rearranging the Visual World for Robotic Manipulation (2020, background): “Alternative representations including keypoints [3, 4, 5, 6] and dense descriptors [9, 10, 11] have shown to be capable of category-level generalization and representing deformable objects, but still struggle to represent scenes with an unknown number of objects (e.”
Self-Supervised Correspondence in Visuomotor Policy Learning (2019, methodology): “For more related work in self-supervised robotic visual learning, including correspondence learning, we refer to the reader to [7].”
kPAM: KeyPoint Affordances for Category-Level Robotic Manipulation (2019, background): “Other recent work has developed dense correspondence visual models, including at a category level, as a general representation for robot manipulation [4], but did not formulate how to specify and solve the task of manipulating objects into specific configurations.”
CLIPort: What and Where Pathways for Robotic Manipulation (2021, background): “Alternatively, dense descriptors [27, 28, 29] and keypoint representations [30, 31, 32] forgo segmentation and pose representations, but do not reason about sequential actions and struggle to represent scenes with variable numbers of objects.”
Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Transporter Networks (2020, background): “This is useful for manipulating deformable objects, where changes between current and desired configurations are better captured with dense correspondences [22], [20].”
Self-Supervised Goal-Conditioned Pick and Place (2020, background): “Another approach is to move objects around in order to learn about their visual appearance [3, 9, 11].”
Form2Fit: Learning Shape Priors for Generalizable Assembly from Disassembly (2019, background): “correspondences is a fundamental task in vision and graphics, studied extensively in prior work via descriptors [13], [14], [15], [16], [17], [18].”
Vision-based robotic grasping from object localization, object pose estimation to grasp estimation for parallel grippers: a review (2020, methodology): “The widely used 3D local shape descriptors, such as Spin Images Johnson (1997), 3D Shape Context Frome et al. (2004), FPFH Rusu et al. (2009), CVFH Aldoma et al.”
Self-supervised 6D Object Pose Estimation for Robot Manipulation (2019, background): “Previous works on self-supervised learning in robot perception mainly focus on learning object segmentation [6], object detection [7], and dense pixel-wise correspondences [29], [30], since it is relatively easy to obtain annotations in 2D image space.”
How to train your robot with deep reinforcement learning: lessons we have learned (2021, methodology): “…methods such as autoencoders (Finn et al. 2016b; Ghadirzadeh et al. 2017), contrastive losses (Sermanet et al. 2018), and correspondence learning (Florence et al. 2018, 2019), provide a reasonable solution in cases where the inductive biases of the unsupervised algorithm effectively match the…”
A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms (2019, background): “hierarchy, and include point cloud, pixel, and voxel representations for capturing the partial or complete shapes of objects in detail [334, 34, 195, 114].”
Keypoints into the Future: Self-Supervised Correspondence in Model-Based Reinforcement Learning (2020, methodology): “Similar to the approach taken in [8, 9], we use visual-correspondence learning, which is trained in a completely self-supervised fashion, to train a visual model which that can be used to ﬁnd correspondences across RGB images.”
Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation (2021, methodology): “To mitigate the generalization issue, other prior work ﬁrst learns to model dense point correspondence in a task-agnostic fashion [10].”
Masked Visual Pre-training for Motor Control (2022, unknown): “Florence et al. (2018) learn dense image descriptors with self-supervision.”
Learning RGB-D Feature Embeddings for Unseen Object Instance Segmentation (2020, background): “Since the concept of “objects“ is quite general, in this work, we consider objects in robotic manipulation environments and focus on segmenting objects in tabletop scenes, which provides useful information for manipulating these objects as shown in [6, 22].”
TAP-Vid: A Benchmark for Tracking Any Point in a Video (2022, unknown): “…bring solutions to many interesting challenges, e.g., better handling of dynamic or deformable objects in SFM [56], and allowing the semantic keypoint-based methods [15, 42, 43, 72] often employed in robotic object manipulation to be applied to generic objects, independent of their class.”
Articulation-Aware Canonical Surface Mapping (2020, background): “Towards this general goal of learning representations that respect correspondence, several prior works attempt to design [22], or learn features invariant to camera movement [9, 38], or synthetic transforms [30].”
Learning Rope Manipulation Policies Using Dense Object Descriptors Trained on Synthetic Depth Data (2020, methodology): “While task agnostic manipulation requires geometric understanding of the objects being manipulated, ﬁne-grained understanding of the object conﬁguration is often unnecessary to effectively grasp or push an object [7, 15, 19, 22–24].”