Joint Training of a Convolutional Network and a Graphical Model for Human Pose Estimation

By Jonathan Tompson, Arjun Jain, Yann LeCun, C. Bregler. 2014

1,614 citations recorded by the source corpus.

This paper proposes a new hybrid architecture that consists of a deep Convolu-tional Network and a Markov Random Field. We show how this architecture is successfully applied to the challenging problem of articulated human pose estimation in monocular images. The architecture can exploit structural domain constraints such as geometric relationships between body joint locations. We show that joint training of these two model paradigms improves performance and allows us to significantly outperform existing state-of-the-art techniques.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

Fully convolutional networks for semantic segmentation (2014, methodology): “[35] to learn an end-to-end part detector and spatial model for pose estimation, although they do not exposit on or analyze this method.”
Spatial Transformer Networks (2015, background): “ the Convolutional Neural Network (CNN) [21]. Though not a recent invention, we now see a cornucopia of CNN-based models achieving state-of-the-art results in classiﬁcation [19, 28, 35], localisation [31, 37], semantic segmentation [24], and action recognition [12, 32] tasks, amongst others. A desir”
Deep Structured Output Learning for Unconstrained Text Recognition (2014, background): “Another example is the recent work of Tompson et al. (2014) for human pose estimation, where an MRF-like model over the distribution of spatial locations for each body part is constructed, incorporating a single round of message-passing.”
Hypercolumns for object segmentation and fine-grained localization (2014, methodology): “Features based on convolutional networks (CNNs) [29] have now led to the best results on a range of vision tasks: image classification [28, 36], object segmentation and detection [18, 22], action classification [35], pose estimation [37] and fine-grained category recognition [44, 6].”
Beyond Skip Connections: Top-Down Modulation for Object Detection (2016, methodology): “After the resurgence of ConvNets for image classification [8, 29], they have been successfully adopted for a variety of computer vision tasks such as object detection [16, 17, 41, 48], semantic segmentation [3, 6, 34, 35], instance segmentation [21, 22, 38], pose estimation [49, 50], depth estimatio”
Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields (2016, methodology): “Tompson et al. [35] used a deep architecture with a graphical model whose parameters are learned jointly with the network.”
Convolutional Pose Machines (2016, methodology): “In summary, our method improves the accuracy in all parts, over all precisions, across all view angles, and is the first one achieving such high accuracy without any pre-training from other data, or post-inference parsing with hand-design priors or initialization of such a structured prediction task”
Deformable part models are convolutional neural networks (2014, neutral): “, [3, 6, 22, 38, 45]).”
End-to-End Training of Deep Visuomotor Policies (2015, methodology): “…exceedingly challenging vision tasks suggests that this class of models is capable of learning invariance to irrelevant distractor features [8, 16, 40], and in principle this issue can be addressed by training the policy in a variety of environments, though this poses certain logistical…”
Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs (2014, background): “While this invariance is clearly desirable for high-level vision tasks, it can hamper low-level tasks, such as pose estimation (Chen & Yuille, 2014; Tompson et al., 2014) and semantic segmentation - where we want precise localization, rather than abstraction of spatial details.”
Conditional Random Fields as Recurrent Neural Networks (2015, methodology): “[57] showed the use of joint training of a CNN and an MRF for human pose estimation, while Chen et al.”
Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-scale Convolutional Architecture (2014, neutral): “However, ConvNets have recently been applied to a variety of other tasks, including pose estimation [36, 27], stereo depth [38, 25], and instance segmentation [14].”
Efficient Piecewise Training of Deep Structured Models for Semantic Segmentation (2015, neutral): “The work in [45] combines CRFs and CNNs for human pose estimation.”
Stacked Hourglass Networks for Human Pose Estimation (2016, methodology): “[15] instead generates heatmaps by running an image through multiple resolution banks in parallel to simultaneously capture features at a variety of scales.”
Fully Connected Deep Structured Networks (2015, background): “[32] propose joint training for pose estimation based on a heuristic approximation which ignores the normalization constant of the model distribution.”
Learning Deep Structured Models (2014, background): “Common approaches to incorporate non-linearities often rely on a two-step process where a non-linear classifier is trained first and its output is used to generate features for the structured predictor [20, 32].”
Wide & Deep Learning for Recommender Systems (2016, methodology): “Joint training of neural networks with graphical models has also been applied to human pose estimation from images [6].”
Recent advances in convolutional neural networks (2015, methodology): “is a Siamese network [150], is jointly trained in cascade with the off-the-shelf model [149].”