Learning Models for Actions and Person-Object Interactions with Transfer to Question Answering

By Arun Mallya, Svetlana Lazebnik. 2016

121 citations recorded by the source corpus.

This paper proposes deep convolutional network models that utilize local and global context to make human activity label predictions in still images, achieving state-of-the-art performance on two recent datasets with hundreds of labels each. We use multiple instance learning to handle the lack of supervision on the level of individual person instances, and weighted loss to handle unbalanced training data. Further, we show how specialized features trained on these datasets can be used to improve accuracy on the Visual Question Answering (VQA) task, in the form of multiple choice fill-in-the-blank questions (Visual Madlibs). Specifically, we tackle two types of questions on person activity and person-object relationship and show improvements over generic features trained on the ImageNet classification task

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Multimodal Explanations: Justifying Decisions and Pointing to the Evidence (2018, background): “Recent work on activity recognition in still images relies on a variety of cues, such as pose and global context [15, 23, 26].”
Detecting and Recognizing Human-Object Interactions (2017, background): “Accurate recognition of human-object interaction can benefit numerous tasks in computer vision, such as actionspecific image retrieval [26], caption generation [35], and question answering [35, 24].”
Attentive Explanations: Justifying Decisions and Pointing to the Evidence (2016, background): “Recent work on activity recognition in still images relies on a variety of cues, such as pose and global context [27, 24].”
HAKE: Human Activity Knowledge Engine (2019, unknown): “Method mAP AlexNet+SVM [13] 19.4 R*CNN [7] 28.5 Girdhar & Ramanan [6] 34.6 Mallya & Lazebnik [19] 36.1 Pairwise [14] 39.9 Pairwise [14]+ Table 3.”
Learning Human-Object Interactions by Graph Parsing Neural Networks (2018, methodology): “[29] modified Fast RCNN model [13] for HOI recognition, with the assistance of Visual Question Answering (VQA).”
The VQA-Machine: Learning How to Use Existing Vision Algorithms to Answer New Questions (2016, background): “However, these modiﬁcations are either focused on developing more advanced embedding techniques or employing different question encoding methods, with only very few meth-ods actually aim to improve the visual information available [21, 30, 35].”
iCAN: Instance-Centric Attention Network for Human-Object Interaction Detection (2018, methodology): “As shown in Figure 2, examples of encoding context include selecting a secondary box [13], using the union of the human and object bounding boxes [29], extracting features around human pose keypoints [6], or exploiting global context from the whole image [31].”
A Comprehensive Survey of Vision-Based Human Action Recognition Methods (2019, methodology): “[120] calculated human and object features from pretrained CNNs, and merged them with pretrained human and object relationship models learned from the HIOC dataset to detect interactions.”
Straight to the Facts: Learning Knowledge Base Retrieval for Factual Visual Question Answering (2018, methodology): “For detecting actions, we use the CNN model proposed in [69] which is trained on the HICO [70] and MPII [71] datasets.”
Holistic++ Scene Understanding: Single-View 3D Holistic Scene Parsing and Human Pose Estimation With Human-Object Interaction and Physical Commonsense (2019, background): “In robotics and computer vision, prior work has exploited human-object relations in event, object, and scene modeling, but most work focuses on human-object relation detection in images [5, 30, 25, 21], probabilistic modeling from multiple data sources [45, 33, 13], and snapshots generation or scene”
Attentional Pooling for Action Recognition (2017, methodology): “Mallya and Lazeb-nik [30] improve upon it by using the full image as the context and using multiple instance learning (MIL) to reason over all humans present in the image to predict an action label for the image.”