Detecting and Recognizing Human-Object Interactions

By Georgia Gkioxari, Ross B. Girshick, Piotr Doll'ar, Kaiming He. 2017

649 citations recorded by the source corpus.

To understand the visual world, a machine must not only recognize individual object instances but also how they interact. Humans are often at the center of such interactions and detecting human-object interactions is an important practical and scientific problem. In this paper, we address the task of detecting (human, verb, object) triplets in challenging everyday photos. We propose a novel model that is driven by a human-centric approach. Our hypothesis is that the appearance of a person - their pose, clothing, action - is a powerful cue for localizing the objects they are interacting with. To exploit this cue, our model learns to predict an action-specific density over target object locations based on the appearance of a detected person. Our model also jointly learns to detect people and objects, and by fusing these predictions it efficiently infers interaction triplets in a clean, jointly trained end-to-end system we call InteractNet. We validate our approach on the recently introduced Verbs in COCO (V-COCO) and HICO-DET datasets, where we show quantitatively compelling results.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

Visual Entailment: A Novel Task for Fine-Grained Image Understanding (2019, background): “Relationship detection among image constituents uses separate branches in a Con-vNet to model objects, humans, and their interactions [5, 21].”
The Open Images Dataset V4 (2018, background): “Recently, high-performing models based on deep convolutional neural networks are dominating the field (Gupta and Malik, 2015; Gkioxari et al., 2018; Gao et al., 2018; Kolesnikov et al., 2018).”
Unidentified Video Objects: A Benchmark for Dense, Open-World Segmentation (2021, background): “Many real-world applications such as object search [27, 30], instance registration [50], human-object interaction modeling [14] and human activity understanding [6] require such open-world prediction abilities, e.”
Understanding Human Hands in Contact at Internet Scale (2020, methodology): “Recent work has largely taken the approach of recognizing verb-noun pairs [17, 9, 10, 19, 6].”
A Metaverse: Taxonomy, Components, Applications, and Open Challenges (2022, neutral): “Some agents classify instructions into a single skill level by mimicking human behavior [156].”
MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domain (2022, methodology): “HOI detection models mostly consider one single object involved in the interaction [43, 42, 39, 99, 8].”
H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactions (2019, background): “While action recognition is a longstanding problem in computer vision [3, 10, 18, 26, 31, 38, 69], first-person action recognition started to emerge as an active field only recently, largely due to the advent of consumer-level wearable sensors and large egocentric datasets [9, 20, 44, 52].”
Ego-Topo: Environment Affordances From Egocentric Video (2020, background): “As a result, scene understanding is largely about answering the who/where/what questions of recognition: what objects are present? is it an indoor/outdoor scene? where is the person and what are they doing? [55, 52, 72, 42, 73, 34, 69, 18].”
Learning Human-Object Interactions by Graph Parsing Neural Networks (2018, methodology): “Previous studies for HOI achieved good performance in their specific domains (spatial [1, 14] or temporal [20,34,35]).”
Pose Flow: Efficient Online Pose Tracking (2018, background): “Furthermore, human pose trajectory extracted from the entire video is a high-level human behavior representation [20, 21], naturally providing us with a powerful tool to handle a series of visual understanding tasks, such as Action Recognition [5], Person Re-identification [19, 24], Human-Object Int”
iCAN: Instance-Centric Attention Network for Human-Object Interaction Detection (2018, methodology): “Following Gkioxari et al. [14] we compute the scores for the triplets in a cascade fashion.”
PPDM: Parallel Point Detection and Matching for Real-Time Human-Object Interaction Detection (2019, background): “Human-Object Interaction (HOI) detection [30, 11, 10, 9, 12, 16, 22] has received increasing attention recently.”
Reconstructing Hand-Object Interactions in the Wild (2020, background): “Examples include detection [11, 15], video recognition [62], interaction hotspots [37], and affordances [38].”
BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation (2022, background): “Researchers have studied to jointly perform object detection and instance segmentation [44, 3] and have extended to pose estimation and human-object interaction [18, 51, 56, 16].”
The MECCANO Dataset: Understanding Human-Object Interactions from Egocentric Videos in an Industrial-like Domain (2020, methodology): “While past works have already investigated the tasks of action recognition [13, 37, 41, 56], active object detection [46], and active object recognition [13] in the context of egocentric vision, Human-Object Interaction (HOI) detection has been generally studied in the context of third person vision”