Learning Human-Object Interactions by Graph Parsing Neural Networks

By Siyuan Qi, Wenguan Wang, Baoxiong Jia, Jianbing Shen, Song-Chun Zhu. European Conference on Computer Vision, 2018

587 citations recorded by the source corpus.

This paper addresses the task of detecting and recognizing human-object interactions (HOI) in images and videos. We introduce the Graph Parsing Neural Network (GPNN), a framework that incorporates structural knowledge while being differentiable end-to-end. For a given scene, GPNN infers a parse graph that includes (i) the HOI graph structure represented by an adjacency matrix, and (ii) the node labels. Within a message passing inference framework, GPNN iteratively computes the adjacency matrices and node labels. We extensively evaluate our model on three HOI detection benchmarks on images and videos: HICO-DET, V-COCO, and CAD-120 datasets. Our approach significantly outperforms state-of-art methods, verifying that GPNN is scalable to large datasets and applies to spatial-temporal settings.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/4ce19e59b6c99b960df645d753fd29d2ccd9b7b4

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

SIGN: Scalable Inception Graph Neural Networks (2020, background): “…modeling relational data in a variety of different domains, including social network link prediction Zhang and Chen (2018), human-object interaction Qi et al. (2018), computer graphics Monti et al. (2016), particle physics Choma et al. (2018), chemistry Duvenaud et al. (2015); Gilmer et al.…”
MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domain (2022, background): “Some works [71, 8, 104] explored HOI detection using graph convolutional neural networks after detecting humans and objects in the scene.”
Holistic++ Scene Understanding: Single-View 3D Holistic Scene Parsing and Human Pose Estimation With Human-Object Interaction and Physical Commonsense (2019, background): “In robotics and computer vision, prior work has exploited human-object relations in event, object, and scene modeling, but most work focuses on human-object relation detection in images [5, 30, 25, 21], probabilistic modeling from multiple data sources [45, 33, 13], and snapshots generation or scene”
An Attention Enhanced Graph Convolutional LSTM Network for Skeleton-Based Action Recognition (2019, methodology): “[19] apply GNN to address the task of detecting and recognizing human-object interactions in images and videos.”
The MECCANO Dataset: Understanding Human-Object Interactions from Egocentric Videos in an Industrial-like Domain (2020, background): “Some works [48, 9, 69] explored HOI detection using graph convolutional neural networks after detecting humans and objects in the scene.”
A Comprehensive Survey of Scene Graphs: Generation and Application (2021, background): “is proposed in [181] to infer the HOI graph structure represented by an adjacency matrix and node labels.”
Zero-Shot Video Object Segmentation via Attentive Graph Neural Networks (2019, methodology): “Recently, GNNs have been successfully applied in many fields, including molecular biology [14], computer vision [48, 71, 76], machine learning [62] and natural language processing [2].”
Deep Learning on Graphs: A Survey (2018, unknown): “…prediction [137], [138], [139], gene expression [140], natural language processing (NLP) [141], [142], computer vision [143], [144], [145], [146], [147], trafﬁc forecasting [148], [149], program induction [150], solving graph-based NP problems [151], [152], and multi-agent AI systems [153],…”
Unified Graph Structured Models for Video Understanding (2021, background): “…note that some approaches which have employed spatio-temporal graphs in video have only considered frame-to-frame interactions [22, 60, 3, 40, 42, 44], rather than long-range relations, and have not evaluated on spatio-temporal localisation which requires more detailed understanding We construct…”
InstaBoost: Boosting Instance Segmentation via Probability Map Guided Copy-Pasting (2019, background): “It plays an vital role in computer vision and has many practical applications in autonomous driving [10], robotic manipulation [38], HOI detection [29, 36] etc.”
Cross-Modal Hierarchical Modelling for Fine-Grained Sketch Based Image Retrieval (2020, methodology): “Graph neural networks (GNN) [3, 37] being well-suited to structured data offer viability towards implementing hierarchy.”
Reasoning Visual Dialogs With Structural and Partial Observations (2019, methodology): “Some of these methods [30, 55, 4, 17, 26, 48] implement every graph node as a small neural network and formulate the interactions between nodes as a message propagation process, which is designed to be end-to-end trainable.”
PPDM: Parallel Point Detection and Matching for Real-Time Human-Object Interaction Detection (2019, methodology): “Human-Object Interaction (HOI) detection [30, 11, 10, 9, 12, 16, 22] has received increasing attention recently.”
Explainability Techniques for Graph Convolutional Networks (2019, background): “…Zitnik et al., 2018), physics (Battaglia et al., 2016; Chang et al., 2017; Gilmer et al., 2017; Watters et al., 2017; Sanchez-Gonzalez et al., 2018), visual recognition (Qi et al., 2017; 2018; Narasimhan et al., 2018), and natural language processing (Bastings et al., 2017; Beck et al., 2018).”
HOTR: End-to-End Human-Object Interaction Detection with Transformers (2021, background): “Graph-based approaches have proposed frameworks that can explicitly represent HOI structures with graphs [24, 26, 4, 28, 21].”
Language-Conditioned Graph Networks for Relational Reasoning (2019, methodology): “Graph networks have been successfully applied to various tasks, from object detection [25] and region clas-siﬁcation [7] to human-object interaction [30] and activity recognition [12].”
Spatio-Temporal Dynamic Inference Network for Group Activity Recognition (2021, background): “Many research areas inherently involve the modeling of interactions like trajectory prediction [16, 38, 34], human object interaction [33, 29, 14] and scene graph generation [44, 49, 7].”