Learning to Detect Human-Object Interactions

By Yu-Wei Chao, Yunfan Liu, Michael Xieyang Liu, Huayi Zeng, Jia Deng. IEEE Workshop/Winter Conference on Applications of Computer Vision, 2017

633 citations recorded by the source corpus.

We study the problem of detecting human-object interactions (HOI) in static images, defined as predicting a human and an object bounding box with an interaction class label that connects them. HOI detection is a fundamental problem in computer vision as it provides semantic information about the interactions among the detected objects. We introduce HICO-DET, a new large benchmark for HOI detection, by augmenting the current HICO classification benchmark with instance annotations. To solve the task, we propose Human-Object Region-based Convolutional Neural Networks (HO-RCNN). At the core of our HO-RCNN is the Interaction Pattern, a novel DNN input that characterizes the spatial relations between two bounding boxes. Experiments on HICO-DET demonstrate that our HO-RCNN, by exploiting human-object spatial relations through Interaction Patterns, significantly improves the performance of HOI detection over baseline approaches.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/60542b1a857024c79db8b5b03db6e79f74ec8f9f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Holistic++ Scene Understanding: Single-View 3D Holistic Scene Parsing and Human Pose Estimation With Human-Object Interaction and Physical Commonsense (2019, background): “In robotics and computer vision, prior work has exploited human-object relations in event, object, and scene modeling, but most work focuses on human-object relation detection in images [5, 30, 25, 21], probabilistic modeling from multiple data sources [45, 33, 13], and snapshots generation or scene”
Visual Entailment: A Novel Task for Fine-Grained Image Understanding (2019, background): “Relationship detection among image constituents uses separate branches in a Con-vNet to model objects, humans, and their interactions [5, 21].”
Attend and Interact: Higher-Order Object Interactions for Video Understanding (2017, background): “Another approach which has been widely used with images is by pairing all possible object candidates (or subject-object pairs) [7, 8, 17, 39, 58].”
HOTR: End-to-End Human-Object Interaction Detection with Transformers (2021, background): “HICO-DET [3] is a subset of HICO dataset and has more than 150K annotated instances of human-object pairs in 47,051 images (37,536 training and 9,515 testing) and is annotated with 600 (cid:104) verb, object (cid:105) interaction types.”
iCAN: Instance-Centric Attention Network for Human-Object Interaction Detection (2018, methodology): “For HICO-DET, we also demonstrate that our method compares favorably against existing methods [4, 14, 35].”
PPDM: Parallel Point Detection and Matching for Real-Time Human-Object Interaction Detection (2019, methodology): “To verify the effectiveness of our PPDM, we conduct experiments not only on our HOI-A dataset but also on the general HOI detection dataset HICO-Det [2].”
Webly Supervised Concept Expansion for General Purpose Vision Models (2022, methodology): “Target outputs are built by gathering all HOI annotations for that input person box (annotations with person boxes that were pruned during de-duplication are mapped to the person box with the highest IoU overlap).”
Knowledge-Embedded Routing Network for Scene Graph Generation (2019, background): “Some other works also attempted to learn human-object interactions [31, 1], in which the subject was a person.”
Reconstructing Hands in 3D with Transformers (2023, methodology): “Among the larger scale efforts, COCO-WholeBody [25] provides hand annotations for the people in the COCO dataset [35] and Halpe [15] annotates hands in the HICO-DET dataset [4, 5].”
Pixels to Graphs by Associative Embedding (2017, neutral): “This includes localization from referential expressions [11], detection of human-object interactions [3], or the more general tasks of visual relationship detection (VRD) [18] and scene graph generation [12].”
HAKE: Human Activity Knowledge Engine (2019, methodology): “HOI [12, 23, 21] usually accounts for a large proportion of daily life activities.”
Learning Human-Object Interactions by Graph Parsing Neural Networks (2018, methodology): “Previous studies for HOI achieved good performance in their specific domains (spatial [1, 14] or temporal [20,34,35]).”
LinkNet: Relational Embedding for Scene Graph (2018, neutral): “This includes detection of human-object interactions [7, 3], localization of proposals from natural language expressions [12], or the more general tasks of visual relationship detection [17, 25, 38, 5, 19, 37, 34, 41] and scene graph generation [33, 18, 35, 22].”
MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domain (2022, background): “HOI detection models mostly consider one single object involved in the interaction [43, 42, 39, 99, 8].”
End-to-End Human Object Interaction Detection with HOI Transformer (2021, methodology): “Training our network takes 7 hours on 8 NVIDIA 2080TI GPU on V-COCO and 70 hours on HICO-DET.”
The MECCANO Dataset: Understanding Human-Object Interactions from Egocentric Videos in an Industrial-like Domain (2020, background): “While past works have already investigated the tasks of action recognition [13, 37, 41, 56], active object detection [46], and active object recognition [13] in the context of egocentric vision, Human-Object Interaction (HOI) detection has been generally studied in the context of third person vision”
QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information (2021, methodology): “In the HICO-DET evaluation, the object class is also taken into account for the judgment.”
Detecting and Recognizing Human-Object Interactions (2017, methodology): “In addition, we prove the effectiveness of InteractNet by reporting a 27% relative improvement on the newly released HICO-DET dataset [3].”
Understanding Human Hands in Contact at Internet Scale (2020, methodology): “ion, and understanding the affordances (opportunities for interaction [16]) has been a long-term interest of computer vision. Recent work has largely taken the approach of recognizing verb-noun pairs [17,9,10,19,6]. In terms of technical approach, our method is most related to the approach of Gkioxa”