Flowing ConvNets for Human Pose Estimation in Videos

By Tomas Pfister, James Charles, Andrew Zisserman. 2015

583 citations recorded by the source corpus.

The objective of this work is human pose estimation in videos, where multiple frames are available. We investigate a ConvNet architecture that is able to benefit from temporal context by combining information across the multiple frames using optical flow. To this end we propose a network architecture with the following novelties: (i) a deeper network than previously investigated for regressing heatmaps, (ii) spatial fusion layers that learn an implicit spatial model, (iii) optical flow is used to align heatmap predictions from neighbouring frames, and (iv) a final parametric pooling layer which learns to combine the aligned heatmaps into a pooled confidence map. We show that this architecture outperforms a number of others, including one that uses optical flow solely at the input layers, one that regresses joint coordinates directly, and one that predicts heatmaps without spatial fusion. The new architecture outperforms the state of the art by a large margin on three video pose estimation datasets, including the very challenging Poses in the Wild dataset, and outperforms other deep methods that don't use a graphical model on the single-image FLIC benchmark (and also [5, 35] in the high precision region).

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields (2016, background): “These top-down approaches directly leverage existing techniques for single-person pose estimation [17, 31, 18, 28, 29, 7, 30, 5, 6, 20], but suffer from early commitment: if the person detector fails–as it is prone to do when people are in close proximity–there is no recourse to recovery.”
Stacked Hourglass Networks for Human Pose Estimation (2016, neutral): “nd reassess higher order spatial relationships. This is similar to other pose estimations methods that have demonstrated strong performance with multiple iterative stages and intermediate supervision [18,19,30]. Consider the limits of applying intermediate supervision with only the use of a single h”
UberNet: Training a Universal Convolutional Neural Network for Low-, Mid-, and High-Level Vision Using Diverse Datasets and Limited Memory (2016, methodology): “aliency estimation [58, 112], semantic segmentation [14, 27,67], region proposal generation [30,82,85], instance segmentation [21,37,82], pose estimation, part segmentation, and landmark localization [3,15,42,75,81,96,98,104,111], as well as the large body of works around object detection and image ”
Convolutional Neural Fabrics (2016, methodology): “Finally, we analyze the number of response maps, parameters, and activations of fabrics in Section 3.3.”
Human Pose Estimation with Iterative Error Feedback (2015, methodology): “We will use as running example the task of 2D human pose estimation [48, 39, 37, 28], where the goal is to infer the 2D locations of a set of keypoints such as wrists, ankles, etc, from a single RGB image.”
Recurrent Human Pose Estimation (2016, methodology): “An alternative, which it turns out might be better suited to ConvNets, is an indirect prediction by first regressing a heatmap over the image for each keypoint, and then obtaining the keypoint position as a mode in this heatmap [20], [27], [38], [37].”
Deep High-Resolution Representation Learning for Human Pose Estimation (2019, neutral): “video pose estimation and tracking [48, 70], etc.”
Unsupervised Learning using Sequential Verification for Action Recognition (2016, methodology): “We evaluate the Probability of Correct Keypoints (PCK) measure [80] on the wrist and elbow keypoints as used in [81] (Figure 6).”
Structured Prediction of 3D Human Pose with Deep Neural Networks (2016, methodology): “started with 2D pose estimation by learning a regressor from an input image to either directly the pose vectors [32] or the heatmaps encoding 2D joint locations [15, 22, 31].”
Knowledge-Guided Deep Fractal Neural Networks for Human Pose Estimation (2017, methodology): “CNN features are regressed in order to provide joint predictions of the body parts [10], [16], [42].”
Look into Person: Joint Body Parsing & Pose Estimation Network and a New Benchmark (2018, methodology): “Following [36], for each parsing result and corresponding ground truth, we compute the center points of regions to obtain joints represented as heatmaps for training more smoothly.”
Monocular human pose estimation: A survey of deep learning-based methods (2020, unknown): “…2016; Chu et al., 2016, 2017; Ning et al., 2018; Ke et al., 2018; Tang et al., 2018a; Tang and Wu, 2019) (4) Temporal constraint : (Jain et al., 2014; Pﬁster et al., 2015; Luo et al., 2018) (5) Network compression : (Tang et al., 2018b; Debnath et al., 2018; Feng et al., 2019) 2D Multiple…”
OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields (2018, methodology): “[33] further used CNNs to implicitly capture global spatial dependencies by designing networks with”
PifPaf: Composite Fields for Human Pose Estimation (2019, background): “Other intermediate representations have been build on top of 2D pose estimates in the image plane including 3D pose estimates [29], human pose estimation in videos [36] and dense pose estimation [16] that would all profit from improved 2D pose estimates.”
Deep Feature Flow for Video Recognition (2016, methodology): “Optical flow information has been exploited to help vision tasks, such as pose estimation [31].”
Fusing 2D Uncertainty and 3D Cues for Monocular Body Pose Estimation (2016, background): “With the growing availability of large datasets and the advent of Deep Learning, the emphasis has shifted towards using discriminative 2D pose regressors [10, 12, 14, 15, 23, 30, 40, 41, 42, 60, 64, 65] to extract the 2D pose and infer a 3D one from it [8, 18, 66, 69].”