Form2Fit: Learning Shape Priors for Generalizable Assembly from Disassembly

By Kevin Zakka, Andy Zeng, Johnny Lee, Shuran Song. 2019

123 citations recorded by the source corpus.

Is it possible to learn policies for robotic assembly that can generalize to new objects? We explore this idea in the context of the kit assembly task. Since classic methods rely heavily on object pose estimation, they often struggle to generalize to new objects without 3D CAD models or task-specific training data. In this work, we propose to formulate the kit assembly task as a shape matching problem, where the goal is to learn a shape descriptor that establishes geometric correspondences between object surfaces and their target placement locations from visual input. This formulation enables the model to acquire a broader understanding of how shapes and surfaces fit together for assembly – allowing it to generalize to new objects and kits. To obtain training data for our model, we present a self-supervised data-collection pipeline that obtains ground truth object-to-placement correspondences by disassembling complete kits. Our resulting real-world system, Form2Fit, learns effective pick and place strategies for assembling objects into a variety of kits – achieving 90% average success rates under different initial conditions (e.g. varying object and kit poses), 94% success under new configurations of multiple kits, and over 86% success with completely new objects and kits. Code, videos, and supplemental material are available at https://form2fit.github.io1

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

Generalization Through Hand-Eye Coordination: An Action Space for Learning Spatially-Invariant Visuomotor Control (2021, methodology): “Our idea of spatially-invariant action space is inspired by a series of work by Zeng et al. and colleagues [34, 37, 39–42].”
Transporter Networks: Rearranging the Visual World for Robotic Manipulation (2020, methodology): “Form2Fit [24], as mentioned in Related Work, is different from our method in that it does not perform visuo-spatial transporting, but instead uses a matching network.”
TossingBot: Learning to Throw Arbitrary Objects With Residual Physics (2019, background): “As in [38], [39], [42], and [43], we account for different grasping angles by rotating the input heightmap by 16 orientations (multiples of 22.”
CLIPort: What and Where Pathways for Robotic Manipulation (2021, background): “On the other hand, end-to-end perception-to-action models can learn precise sequential policies [2, 4, 6, 33, 34, 35], but these methods have limited understanding of semantic concepts and rely on goal-images to condition policies.”
Clear Grasp: 3D Shape Estimation of Transparent Objects for Manipulation (2019, unknown): “clouds) [50, 60, 54, 57].”
Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Transporter Networks (2020, background): “This technique [67] has shown promising results for robotic warehouse order picking [71], pushing and grasping [70], tossing objects [69], kit assembly [66], and mobile manipulation [62].”
Visual Room Rearrangement (2021, background): “Some works address some of these issues, such as generalization to new objects or imperfect perception [54, 6].”
Spatial Action Maps for Mobile Manipulation (2020, methodology): “Similar approaches are used to predict affordances for grasping or suction in [44, 46, 30, 41].”
Grasping in the Wild: Learning 6DoF Closed-Loop Grasping From Low-Cost Demonstrations (2019, background): “It would be interesting to investigate how to make use of the other information captured in this dataset, such as picking order and placing trajectories [34].”
Self-Supervised Goal-Conditioned Pick and Place (2020, background): “Several prior approaches have shown that robotic grasping data can be used to learn object embeddings useful for goal-conditioned grasping [5] and assembly [12].”