Learning Human Pose Estimation Features with Convolutional Networks

By Arjun Jain, Jonathan Tompson, Mykhaylo Andriluka, Graham W. Taylor, C. Bregler. 2013

218 citations recorded by the source corpus.

Abstract: This paper introduces a new architecture for human pose estimation using a multi- layer convolutional network architecture and a modified learning technique that learns low-level features and higher-level weak spatial models. Unconstrained human pose estimation is one of the hardest problems in computer vision, and our new architecture and learning schema shows significant improvement over the current state-of-the-art results. The main contribution of this paper is showing, for the first time, that a specific variation of deep learning is able to outperform all existing traditional architectures on this task. The paper also discusses several lessons learned while researching alternatives, most notably, that it is possible to learn strong low-level feature detectors on features that might even just cover a few pixels in the image. Higher-level spatial models improve somewhat the overall result, but to a much lesser extent then expected. Many researchers previously argued that the kinematic structure and top-down information is crucial for this domain, but with our purely bottom up, and weak spatial model, we could improve other more complicated architectures that currently produce the best results. This mirrors what many other researchers, like those in the speech recognition, object recognition, and other domains have experienced.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

Towards Accurate Multi-person Pose Estimation in the Wild (2017, methodology): “…architecture trained on whole images (rather than image crops) to infer the heatmap potentials, and they reformulated the graphical model from [26] - simplifying the tree structure to a star-graph and re-writing the belief propagation messages - so that the entire system could be trained…”
Joint Training of a Convolutional Network and a Graphical Model for Human Pose Estimation (2014, methodology): “Similar to Jain et al. [16], we formulate the Spatial-Model as an MRF-like model over the distribution of spatial locations for each body part.”
Recent advances in convolutional neural networks (2015, methodology): “LIC [101] and LSP [102]. Except this holistic manner, some recent works exploit CNN to directly learn representation of local body parts instead of using hand-crafted low-level features. Ajrun et al. [103] present a CNN based end-to-end learning approach for fullbody human pose estimation, in which ”
C3D: Generic Features for Video Analysis (2014, methodology): “ConvNets have been applied to the problem of human pose estimation in both images [17] and videos [18].”
Efficient object localization using Convolutional Networks (2014, methodology): “The current state-of-the-art methods for the task of human-pose estimation in-the-wild are also built using ConvNets [22, 13, 21, 14, 5].”
Part Detector Discovery in Deep Convolutional Neural Networks (2014, neutral): “The work of Jain et al. [20] uses the sliding window approach for part localization with CNNs.”
Flowing ConvNets for Human Pose Estimation in Videos (2015, methodology): “developed ConvNet architectures to directly regress heatmaps for each joint, with subsequent layers to add an Markov Random Field (MRF)-based spatial model [17, 36], and a pose refinement model [35] (based on a Siamese network with shared weights) upon a rougher pose estimator ConvNet.”
MoDeep: A Deep Learning Framework Using Motion Features for Human Pose Estimation (2014, methodology): “Jain et al. [50] do not use multi-scale information and evaluate their model in a sliding window fashion, whereas we use the ‘one-shot’ approach.”
3D Human Pose Estimation from Monocular Images with Deep Convolutional Neural Network (2014, background): “[18] trains convolutional neural networks to classify whether a given window contains one specific body-part, and then detection maps for each body-part are calculated by sliding the detection window over the whole image.”
Efficient ConvNet-based marker-less motion capture in general scenes with a low number of cameras (2015, background): “Toshev et al. [40] formulate the problem as a direct regression to joint location.”
Deep learning for visual understanding (2017, unknown): “[186] trained multiple smaller convnets to perform independent binary body-part classi cation, followed with a higher-level weak spatial model to remove strong outliers and to enforce global pose consistency.”
Sparseness Meets Deepness: 3D Human Pose Estimation from Monocular Video (2015, methodology): “The second approach uses a CNN architecture for body part detection [10, 17, 45, 31] and then typically enforces the 2D spatial relationship between body parts as a subsequent processing step.”
Structured Prediction of 3D Human Pose with Deep Neural Networks (2016, methodology): “started with 2D pose estimation by learning a regressor from an input image to either directly the pose vectors [32] or the heatmaps encoding 2D joint locations [15, 22, 31].”
Deep learning for visual understanding: A review (2016, background): “Moreover, Jain et al. [206] trained multiple smaller convnets to perform independent binary body-part classi ﬁ cation, followed with a higher-level weak spatial model to remove strong outliers and to enforce global pose consistency.”
Learning Spatiotemporal Features with 3D Convolutional Networks (2014, methodology): “ConvNets have also been applied to the problem of human pose estimation in both images [12] and videos [13].”
Cascaded Pyramid Network for Multi-person Pose Estimation (2017, background): “Recent works [41, 40, 8, 30, 15, 5, 21, 20, 1, 42, 44] mostly rely on the development of convolutional neural network(CNN) [28, 17], which largely improve the performance of pose estimation.”
Heterogeneous Multi-task Learning for Human Pose Estimation with Deep Convolutional Neural Network (2014, background): “We also empirically show that the learned neurons in the middle layer of our network are tuned to localized body parts.”
Do Convnets Learn Correspondence? (2014, background): “Recently, Toshev et al. [22] trained a cascade of regression-based convnets for human pose estimation and Jain et al. [23] combine a weak spatial model with deep learning methods.”