Learning task-oriented grasping for tool manipulation from simulated self-supervision

By Kuan Fang, Yuke Zhu, Animesh Garg, Andrey Kurenkov, Viraj Mehta, Li Fei-Fei, Silvio Savarese. 2018

251 citations recorded by the source corpus.

Tool manipulation is vital for facilitating robots to complete challenging task goals. It requires reasoning about the desired effect of the task and, thus, properly grasping and manipulating the tool to achieve the task. Most work in robotics has focused on task-agnostic grasping, which optimizes for only grasp robustness without considering the subsequent manipulation tasks. In this article, we propose the Task-Oriented Grasping Network (TOG-Net) to jointly optimize both task-oriented grasping of a tool and the manipulation policy for that tool. The training process of the model is based on large-scale simulated self-supervision with procedurally generated tool objects. We perform both simulated and real-world experiments on two tool-based manipulation tasks: sweeping and hammering. Our model achieves overall 71.1% task success rate for sweeping and 80.0% task success rate for hammering.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Human-Computer Interaction

Citations in the corpus

IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Data (2019, background): “Recent research has successfully leveraged Reinforcement Learning (RL) for short-horizon robotic manipulation tasks, such as pushing and grasping objects [10, 22, 37].”
ROBOTURK: A Crowdsourcing Platform for Robotic Skill Learning through Imitation (2018, methodology): “Recent research has successfully employed Reinforcement Learning (RL), Self-Supervised Learning (SSL), and Imitation Learning (IL) for short-horizon skill learning, such as pushing and grasping objects [4, 5, 6].”
Dynamics Learning with Cascaded Variational Inference for Multi-Step Manipulation (2019, methodology): “In robot manipulation, several methods define separate grasping and manipulation policies for pushing/tool use [21, 22, 23].”
ManipulaTHOR: A Framework for Visual Object Manipulation (2021, background): “A long-standing problem in robotics research is object manipulation [12, 4, 5, 31, 39, 6, 9, 21, 10].”
Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes (2021, background): “Model-based grasping circumvents reasoning about the physics of contact and grasp generation by pre-defining a set of grasps in the object frame and transform those grasps according to the 6-DoF object pose [1, 2, 3, 4] or detected keypoints of the objects [5, 6].”
Generalization to New Actions in Reinforcement Learning (2020, background): “…been used to transfer policies to new environments (Cobbe et al., 2018; Nichol et al., 2018; Packer et al., 2018), agent morphologies (Wang et al., 2018; Sanchez-Gonzalez et al., 2018; Pathak et al., 2019), and visual inputs for manipulation of unseen tools (Fang et al., 2018; Xie et al., 2019).”
A Joint Network for Grasp Detection Conditioned on Natural Language Commands (2021, background): “3) Semantic Grasp Detection: Semantic grasp detection seeks functionally suitable grasps for specific downstream tasks [23]–[26], such as the identification of different grasps on an object for different purpose.”
Improvisation through Physical Understanding: Using Novel Objects as Tools with Visual Foresight (2019, methodology): “ing of the selected tool [31, 40, 5, 2, 14], and using the grasped tool through planning [29] or policy learning [14].”
KETO: Learning Keypoint Representations for Tool Manipulation (2019, methodology): “a) End-to-End: The End-to-End method is adopted in TOG-Net [16] that directly predicts the manipulation actions”
Grasp'D: Differentiable Contact-rich Grasp Synthesis for Multi-fingered Hands (2022, background): “Visual inputs from the environment are processed in our brain to recognize objects of interest (object recognition [16, 22, 26, 34, 85]), identify modes of interaction to achieve a certain function (affordance prediction [7, 20, 53, 72, 76]), and position our hand(s) in a way that enables that funct”
The Tools Challenge: Rapid Trial-and-Error Learning in Physical Problem Solving (2019, background): “Recent work has also investigated combining imitation learning with model learning and policy updates to learn how to use objects as tools from pixel inputs alone (A. Xie et al., 2019), as well as how to use speciﬁc tools to accomplish new tasks with self-supervised objectives (Fang et al., 2018).”