A Joint Network for Grasp Detection Conditioned on Natural Language Commands
By Yiye Chen, Ruinian Xu, Yunzhi Lin, Patricio A. Vela. 2021
58 citations recorded by the source corpus.
We consider the task of grasping a target object based on a natural language command query. Previous work primarily focused on localizing the object given the query, which requires a separate grasp detection module to grasp it. The cascaded application of two pipelines incurs errors in overlapping multi-object cases due to ambiguity in the individal outputs. This work proposes a model named Command Grasping Network (CGNet) to directly output command satisficing grasps from RGB image and textual command inputs. A dataset with ground truth (image, command, grasps) tuple is generated based on the VMRD dataset to train the proposed network. Experimental results on the generated test set show that CGNet outperforms a cascaded object-retrieval and grasp detection baseline by a large margin. Three physical experiments demonstrate the functionality and performance of CGNet.
Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, Natural Language Processing, Human-Computer Interaction
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Sergey Levine
- Li Fei-Fei
- Luc Van Gool
- Trevor Darrell
- Jitendra Malik
- Dacheng Tao
- Xiaogang Wang
- Andrew Zisserman
In Control and Systems Engineering
- Sergey Levine
- Chelsea Finn
- Pieter Abbeel
- Dieter Fox
- Pierre Sermanet
- Peter R. Florence
- Lucas Manuelli
- Karol Hausman
In Natural Language Processing
- Mohit Shridhar
- Lucas Manuelli
- Brian Ichter
- Dieter Fox
- Fei Xia
- Sergey Levine
- Karol Hausman
- Igor Mordatch
In Human-Computer Interaction
- Mohit Shridhar
- Lucas Manuelli
- Sergey Levine
- Dieter Fox
- Jonathan Tompson
- Pierre Sermanet
- Andy Zeng
- Peter R. Florence
Citations in the corpus
- Inner Monologue: Embodied Reasoning through Planning with Language Models (2022, background): “Various works have investigated strategies for the challenging problem of fusing vision, language, and control [60, 61, 62, 63, 64, 65, 66].”
- CLIPort: What and Where Pathways for Robotic Manipulation (2021, background): “Several works have proposed systems for instructing robots with natural language [7, 8, 9, 10, 41, 42, 43, 44, 45, 46, 47].”
- Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation (2022, methodology): “Several works have proposed methods for grounding language in robot actions [49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60].”