Combining local appearance and holistic view: Dual-Source Deep Neural Networks for human pose estimation

By Xiaochuan Fan, Kang Zheng, Yuewei Lin, Song Wang. 2015

228 citations recorded by the source corpus.

We propose a new learning-based method for estimating 2D human pose from a single image, using Dual-Source Deep Convolutional Neural Networks (DS-CNN). Recently, many methods have been developed to estimate human pose by using pose priors that are estimated from physiologically inspired graphical models or learned from a holistic perspective. In this paper, we propose to integrate both the local (body) part appearance and the holistic view of each local part for more accurate human pose estimation. Specifically, the proposed DS-CNN takes a set of image patches (category-independent object proposals for training and multi-scale sliding windows for testing) as the input and then learns the appearance of each local part by considering their holistic views in the full body. Using DS-CNN, we achieve both joint detection, which determines whether an image patch contains a body joint, and joint localization, which finds the exact location of the joint in the image patch. Finally, we develop an algorithm to combine these joint detection/localization results from all the image patches for estimating the human pose. The experimental results show the effectiveness of the proposed method by comparing to the state-of-the-art human-pose estimation methods based on pose priors that are estimated from physiologically inspired graphical models or learned from a holistic perspective.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Recent advances in convolutional neural networks (2015, methodology): “There is also a pose estimation method called dual-source CNN [153] that integrates graphical models and holistic style.”
Space-Time Representation of People Based on 3D Skeletal Data: A Review (2016, background): “Although recent works based on deep learning [66], [67], [70] showed promising skeleton tracking results,”
Recurrent Human Pose Estimation (2016, neutral): “7 57.1 Wang Li et al. [37] 84.7 57.1 43.7 36.7 56.7 52.4 50.8 54.6 Carreira et al. [8] 90.5 81.8 65.8 59.8 81.6 70.6 62.0 73.1 Chen & Yuille [9] 91.8 78.2 71.8 65.5 73.3 70.2 63.4 73.4 Fan et al. [12] 92.4 75.2 65.3 64.0 75.7 68.3 70.4 73.0 Pishchulin et al. [24] 97.0 91.0 83.8 78.1 91.0 86.7 82”
Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue (2016, background): “Trained initially on tasks such as image classiﬁcation, and ﬁne-tuned to ﬁt other tasks, supervised CNNs are now state of the art for object detection [11], per-pixel image classiﬁcation [19], depth and normal prediction from single image [15], human pose estimation [7] and many other applications.”
Deep High-Resolution Representation Learning for Human Pose Estimation (2019, background): “Nowadays, deep convolutional neural network provides dominant solutions [20, 34, 60, 41, 42, 47, 56, 16].”
LCR-Net++: Multi-Person 2D and 3D Pose Detection in Natural Images (2018, background): “Most state-of-the-art approaches for 2D human pose estimation employ CNN architectures [1], [2], [3], [13], [17], [18], [19], [20].”
Human Pose Estimation via Convolutional Part Heatmap Regression (2016, unknown): “ 90.6 78.1 73.8 68.8 74.8 69.9 58.9 73.6 Carreira et al., CVPR’16 [27] 90.5 81.8 65.8 59.8 81.6 70.6 62.0 73.1 Tompson et al., NIPS’14 [23] 90.6 79.2 67.9 63.4 69.5 71.0 64.2 72.3 Fan et al., CVPR’15 [41] 92.4 75.2 65.3 64.0 75.7 68.3 70.4 73.0 Chen&Yuille, NIPS’14 [22] 91.8 78.2 71.8 65.5 73.3 ”
Deep Learning-based Human Pose Estimation: A Survey (2020, methodology): “, [18, 55, 117, 119, 148, 149, 154, 155, 173, 179, 191, 223, 234, 285]) to predict joint coordinates from images as shown in Fig.”
Associative Embedding: End-to-End Learning for Joint Detection and Grouping (2016, background): “Multiperson Pose Estimation Recent methods have made great progress improving human pose estimation in images in particular for single person pose estimation [50, 48, 52, 40, 8, 5, 41, 4, 14, 19, 34, 26, 7, 49, 44].”
Pose-Invariant Embedding for Deep Person Re-Identification (2017, background): “-scale features and study mechanisms on how to combine them [29,26]. It is also effective to inject spatial relationships between body joints by regularizing the unary scores and pairwise comparisons [11,27]. This paper adopts the convolutional pose machines (CPM) [34], a state-of-the-art pose estim”
Survey on Emotional Body Gesture Recognition (2018, methodology): “Dual-source CNN (DS-CNN) [77] is another framework which can be utilized in the context of HPE.”
Distribution-Aware Coordinate Representation for Human Pose Estimation (2019, methodology): “But only a handful of existing methods adopt this design (Toshev and Szegedy 2014; Fan et al. 2015; Carreira et al. 2016; Sun et al. 2018).”
Stacked Hourglass Networks for Human Pose Estimation (2016, background): “Others have recently tackled this in similar ways [17,20,25] with variations on how to approach unary score generation and pairwise comparison of adjacent joints.”
3D Human Pose Estimation Using Convolutional Neural Networks with 2D Pose Information (2016, background): “Xiaochuan et al. [14] integrated both the local part appearance and the holistic view of an image using dual-source CNN. Convolu-3D tional pose machine [3] is a systematic approach to improve prediction of each stage.”
Deep learning for visual understanding (2017, unknown): “[190] proposed a dual-source convolutional neutral network (DS-CNN) to integrate the holistic and partial view in the CNN framework.”