R-CNNs for Pose Estimation and Action Detection

By Georgia Gkioxari, Bharath Hariharan, Ross B. Girshick, Jitendra Malik. 2014

170 citations recorded by the source corpus.

We present convolutional neural networks for the tasks of keypoint (pose) prediction and action classification of people in unconstrained images. Our approach involves training an R-CNN detector with loss functions depending on the task being tackled. We evaluate our method on the challenging PASCAL VOC dataset and compare it to previous leading approaches. Our method gives state-of-the-art results for keypoint and action prediction. Additionally, we introduce a new dataset for action detection, the task of simultaneously localizing people and classifying their actions, and present results using our approach.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Hypercolumns for object segmentation and fine-grained localization (2014, background): “mance to [20], the previous best on this dataset.”
Cross-Stitch Networks for Multi-task Learning (2016, methodology): “To limit the scope of this paper, we only consider tasks which take the same single input, e.g ., an image as opposed to say an image and a depth-map [25].”
Viewpoints and keypoints (2014, methodology): “We refer the reader to [13] for further details on the evaluation metrics for the task of articulated pose estimation.”
Visual Semantic Role Labeling (2015, background): “[12] extended the dataset for action detection where the task is to detect and localize people doing actions.”
Learning to push by grasping: Using multiple tasks for effective learning (2016, methodology): “Multi task learning (MTL) is generally used to model related tasks closely [24], [25], [26] by using a shared representation and exploiting the commonality and structure of these tasks.”
Monocular human pose estimation: A survey of deep learning-based methods (2020, methodology): “…deep learning brings a huge impact on vision-based human pose estimation, traditional 2D HPE algorithms adopt hand-craft feature extraction and sophisticated body models to obtain local representations and global pose structures (Dantone et al., 2013; Chen and Yuille, 2014; Gkioxari et al., 2014b).”
Region-Based Convolutional Networks for Accurate Object Detection and Segmentation (2016, background): “Gkioxari et al. [44] use multi-task learning to train R-CNNs for person detection, 2D pose estimation, and action recognition.”
End-To-End Multi-Task Learning With Attention (2018, methodology): “In computer vision, multi-task learning has been used to for learning similar tasks such as image classification in multiple domains [23], pose estimation and action recognition [9], and dense prediction of depth, surface normals, and semantic classes [20, 7].”
GPLAC: Generalizing Vision-Based Robotic Skills Using Weakly Labeled Images (2017, background): “A standard approach is to train convolutional neural networks with some layers shared across tasks, and some layers speciﬁc to the individual tasks [13, 51, 56, 17].”
Synthesizing Training Images for Boosting Human 3D Pose Estimation (2016, methodology): “Both Tompson et al. [47] and Chen and Yuille [9] proposed to represent the spatial relationships between joints by graphical models, while training a CNN for predicting 2D joint positions.”
Beyond Bounding Boxes: Precise Localization of Objects in Images (2015, background): “[27] finetuned a network for pose, person detection and action classification, and then trained an SVM to assign a score to the keypoint predictions.”
LCR-Net: Localization-Classification-Regression for Human Pose (2017, methodology): “Most similar to our approach are the methods that jointly localize humans and estimate their 2D pose [10, 11, 21].”
Convolutional Models for Joint Object Categorization and Pose Estimation (2015, background): “Recently (Gkioxari et al., 2014) proposed joint optimization on human pose and activity.”
LCR-Net: Localization-Classiﬁcation-Regression for Human Pose (unknown): “Most similar to our approach are the methods that jointly localize humans and estimate their 2D pose [10, 11, 21].”
What Do I Annotate Next? An Empirical Study of Active Learning for Action Localization (2018, background): “Many techniques have been developed over the years to recognize [11,12,49,59,67], and localize human activities, either in images [28,47,73] or videos [29,34,69].”
Cross-Domain Self-Supervised Multi-task Feature Learning Using Synthetic Imagery (2017, background): “Multi-task learning Multi-task learning [10] has been used for a variety of vision problems including surface normal and depth prediction [16, 17], semantic segmentation [45], pose estimation [24], robot manipulation [55, 54], and face detection [77].”
Convolutional Channel Features (2015, background): “…finetuning, CNN features learned on large-scale recognition tasks have been successfully transferred to various vision tasks like object detection [21], semantic segmentation [36] and action detection [23], but they suffer from heavy computation cost caused by high-level connections [16, 20].”
Robust Optimization for Deep Regression (2015, methodology): “The regularization of the L 2 loss function has been also addressed by Gkioxari et al. [14], where the function being minimized comprises a body pose estimation term (based on L 2 norm) and an action detection term.”