Contextual Action Recognition with R*CNN

By Georgia Gkioxari, Ross B. Girshick, Jitendra Malik. 2015

419 citations recorded by the source corpus.

There are multiple cues in an image which reveal what action a person is performing. For example, a jogger has a pose that is characteristic for jogging, but the scene (e.g. road, trail) and the presence of other joggers can be an additional source of information. In this work, we exploit the simple observation that actions are accompanied by contextual cues to build a strong action recognition system. We adapt RCNN to use more than one region for classification while still maintaining the ability to localize the action. We call our system R*CNN. The action-specific models and the feature maps are trained jointly, allowing for action specific representations to emerge. R*CNN achieves 90.2% mean AP on the PASAL VOC Action dataset, outperforming all other approaches in the field by a significant margin. Last, we show that R*CNN is not limited to action recognition. In particular, R*CNN can also be used to tackle fine-grained tasks such as attribute classification. We validate this claim by reporting state-of-the-art performance on the Berkeley Attributes of People dataset.

Subfields: Artificial Intelligence, Biomedical Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

What makes ImageNet good for transfer learning? (2016, methodology): “For PASCAL-ACT-CLS, we used PASCAL VOC 2012 train/val for finetuning and testing using the experimental setup and code provided by R*CNN [17].”
Relation Networks for Object Detection (2017, background): “Human centered scenarios Quite a few works focus on human-object relation [51, 22, 20, 21].”
Spatial Transformer Networks (2015, background): “cent invention, we now see a cornucopia of CNN-based models achieving state-of-the-art results in classiﬁcation [19, 28, 35], localisation [31, 37], semantic segmentation [24], and action recognition [12, 32] tasks, amongst others. A desirable property of a system which is able to reason about image”
Beyond Skip Connections: Top-Down Modulation for Object Detection (2016, background): “, using other objects [20] and other regions [18] as context.”
Multi-task Self-Supervised Visual Learning (2017, background): “Several recent works have trained deep visual representations us-ing multiple tasks [9, 12, 22, 37], including one work [18] which combines no less than 7 tasks.”
Women also Snowboard: Overcoming Bias in Captioning Models (2018, background): “Exploiting contextual cues can frequently lead to better performance on computer vision tasks [1,2,3].”
UberNet: Training a Universal Convolutional Neural Network for Low-, Mid-, and High-Level Vision Using Diverse Datasets and Limited Memory (2016, background): “…and classiﬁcation, [25] propose a network that jointly solves surface normal estimation, depth estimation and semantic segmentation, while [33] train a system for joint detection, pose estimation and region proposal generation; [70] study the effects of sharing information across networks…”
Recent advances in convolutional neural networks (2015, methodology): “CNN based representation of contextual information is utilized for action recognition in [178].”
Multimodal Explanations: Justifying Decisions and Pointing to the Evidence (2018, background): “Recent work on activity recognition in still images relies on a variety of cues, such as pose and global context [15, 23, 26].”
Detecting and Recognizing Human-Object Interactions (2017, background): “, detecting objects [10, 9, 27] and estimating human actions/poses [12, 32, 2], has witnessed significant improvements thanks to deep learning visual representations [18, 30, 31, 17].”
HAKE: Human Activity Knowledge Engine (2019, neutral): “Method mAP AlexNet+SVM [13] 19.4 R*CNN [7] 28.5 Girdhar & Ramanan [6] 34.6 Mallya & Lazebnik [19] 36.1 Pairwise [14] 39.9 Pairwise [14]+ Table 3.”
Attentional Pooling for Action Recognition (2017, methodology): “[18] modiﬁed R-CNN pipeline to propose R*CNN, where they choose an auxiliary box to encode context apart from the human bounding box.”
Learning Spatial Regularization with Image-Level Supervisions for Multi-label Image Classification (2017, methodology): “For WIDER-Attribute dataset, RCNN [8], R*CNN [9], and DHC [24] are compared.”
Exploring Person Context and Local Scene Context for Object Detection (2015, background): “[11] automatically pick contextual regions for action classification.”
Spatial Memory for Context Reasoning in Object Detection (2017, background): “The second type models object-object relationships at instance-level [15, 27, 32, 58, 67, 94].”
Detecting Visual Relationships with Deep Relational Networks (2017, methodology): “In most of these studies, relationships are usually extracted using simple heuristics or hand-crafted features, and used as an auxiliary components to facilitate other tasks, such as object recognition [25–33], image classiﬁcation and retrieval [34,35], scene understanding 1 code available at…”
Multi-region Two-Stream R-CNN for Action Detection (2016, positive): “[16] boost R-CNN based action detection in still images by adding contextual features.”
The Role of Typicality in Object Classification: Improving The Generalization Capacity of Convolutional Neural Networks (2016, background): “…Zisserman, 2015], object detection [Ren et al. , 2015], image retrieval [Sharif Razavian et al. , 2015], ﬁne-grained recognition [Lin et al. , 2015], scene classiﬁcation [Zhou et al. , 2014], action classiﬁcation [Gkioxari et al. , 2015] and generating image descriptions [Vinyals et al. , 2014].”
Robust Image Completion via Deep Feature Transformations (2019, background): “INTRODUCTION Over the past decades, significant progress has been achieved to develop new deep learning networks that achieve improved solutions for a range of computer vision problems, including image classification [1]–[3], face recognition [1], [4], [5], object detection [6]–[8], semantic segment”