AffordanceNet: An End-to-End Deep Learning Approach for Object Affordance Detection

By Thanh-Toan Do, Anh Viet Nguyen, Ian David Reid, Darwin G. Caldwell, N. Tsagarakis. IEEE International Conference on Robotics and Automation, 2017

366 citations recorded by the source corpus.

We propose AffordanceNet, a new deep learning approach to simultaneously detect multiple objects and their affordances from RGB images. Our AffordanceNet has two branches: an object detection branch to localize and classify the object, and an affordance detection branch to assign each pixel in the object to its most probable affordance label. The proposed framework employs three key components for effectively handling the multiclass problem in the affordance mask: a sequence of deconvolutional layers, a robust resizing strategy, and a multi-task loss function. The experimental results on the public datasets show that our AffordanceNet outperforms recent state-of-the-art methods by a fair margin, while its end-to-end architecture allows the inference at the speed of 150ms per image. This makes our AffordanceNet well suitable for real-time robotic applications. Furthermore, we demonstrate the effectiveness of AffordanceNet in different testing environments and in real robotic applications. The source code is available at https://github.com/nqanh/affordance-net.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, Industrial and Manufacturing Engineering

View Paper: https://www.semanticscholar.org/paper/5d3b74d0702bbdca96f326d2d0ba96349f5c42a6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Kristen Grauman
2. Luc Van Gool
3. Jitendra Malik
4. Trevor Darrell
5. Li Fei-Fei
6. Cordelia Schmid
7. Andrew Zisserman
8. Dacheng Tao

In Control and Systems Engineering

1. Yuke Zhu
2. Silvio Savarese
3. Li Fei-Fei
4. Kristen Grauman
5. Animesh Garg
6. Tushar Nagarajan
7. Xiaolong Wang
8. Francesc Moreno-Noguer

In Industrial and Manufacturing Engineering

1. Xiaolong Wang
2. Francesc Moreno-Noguer
3. Kristen Grauman
4. Yuke Zhu
5. Christoph Feichtenhofer
6. Li Fei-Fei

Citations in the corpus

• Coarse-to-Fine Reasoning for Visual Question Answering (2021, methodology): “For image representation, grid features [23, 65] or object features [2,13,43,46,49] are widely used.”
• Hand-Object Contact Consistency Reasoning for Human Grasps Generation (2021, background): “Predicting scene and object affordance plays an important role in visual understanding [27, 52, 6, 10, 53, 12, 25, 15, 51].”
• Learning Dexterous Grasping with Object-Centric Visual Affordances (2020, background): “Traditionally, supervision comes from labeled image examples [32], [9] or a robot’s grasp success/failure [40], [22], [23], while newer work explores weaker modes of supervision from”
• Video Object Segmentation using Teacher-Student Adaptation in a Human Robot Interaction (HRI) Setting (2018, methodology): “Video semantic segmentation for robotics is widely used in different applications such as autonomous driving [4][24], and robot manipulation [5][10].”
• Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videos (2022, background): “Object affordance grounding [19,21,42,60,69,74] refers to locating where the interaction occurs on an object.”
• KETO: Learning Keypoint Representations for Tool Manipulation (2019, methodology): “To enable robots to understand and manipulate tools, several pioneer works focus on predicting affordance and functional regions on tool objects [30], [48], [41], [58], [27], [14].”
• GanHand: Predicting Human Grasp Affordances in Multi-Object Scenes (2020, background): “Recent works, address this problem using deep learning [36, 53, 17, 6, 59].”
• Vision-based robotic grasping from object localization, object pose estimation to grasp estimation for parallel grippers: a review (2020, background): “Robotic affordances [Do et al., 2018b; Ardón et al., 2019; Chu et al., 2019] usu-”
• Grounded Human-Object Interaction Hotspots From Video (2018, methodology): “They assume access to training images with manually drawn masks or keypoints [45, 10, 12] and some leverage additional sensors like depth [31, 65, 66] or force gloves [3], all of which restrict scalability.”
• Grasp'D: Differentiable Contact-rich Grasp Synthesis for Multi-fingered Hands (2022, background): “Visual inputs from the environment are processed in our brain to recognize objects of interest (object recognition [16, 22, 26, 34, 85]), identify modes of interaction to achieve a certain function (affordance prediction [7, 20, 53, 72, 76]), and position our hand(s) in a way that enables that funct”