Human Pose Estimation with Iterative Error Feedback

By João Carreira, Pulkit Agrawal, Katerina Fragkiadaki, Jitendra Malik. 2015

812 citations recorded by the source corpus.

Hierarchical feature extractors such as Convolutional Networks (ConvNets) have achieved impressive performance on a variety of classification tasks using purely feedforward processing. Feedforward architectures can learn rich representations of the input space but do not explicitly model dependencies in the output spaces, that are quite structured for tasks such as articulated human pose estimation or object segmentation. Here we propose a framework that expands the expressive power of hierarchical feature extractors to encompass both input and output spaces, by introducing top-down feedback. Instead of directly predicting the outputs in one go, we use a self-correcting model that progressively changes an initial solution by feeding back error predictions, in a process we call Iterative Error Feedback (IEF). IEF shows excellent performance on the task of articulated pose estimation in the challenging MPII and LSP benchmarks, matching the state-of-the-art without requiring ground truth scale annotation.

Subfields: Artificial Intelligence, Biomedical Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

What makes ImageNet good for transfer learning? (2016, methodology): “Using ImageNet pre-trained CNN features, impressive results have been obtained on several image classification datasets [13, 35], object detection [16, 39], action recognition [41], human pose estimation [9], image segmentation [10], optical flow [47], image captioning [12, 22] and others [26].”
Beyond Skip Connections: Top-Down Modulation for Object Detection (2016, background): “Contemporary works [5, 12, 32, 43], which incorporate top-down feedback loop in ConvNets, have used ‘unrolling’ to train these networks stagewise.”
Convolutional Pose Machines (2016, background): “[6] train a deep network that iteratively improves part detections using error feedback but use a cartesian representation as in [40] which does not preserve spatial uncertainty and results in lower accuracy in the highprecision regime.”
Perceiver: General Perception with Iterative Attention (2021, background): “Re-entrant processing of this kind (sometimes referred to as top-down processing) has a long history in computer vision (Borenstein et al., 2004; Kumar et al., 2005; Carreira et al., 2016; Hu & Ramanan, 2016; Yang et al., 2018; Lin et al., 2020).”
Deep Clustering for Unsupervised Learning of Visual Features (2018, background): “Pre-trained convolutional neural networks, or convnets, have become the building blocks in most computer vision applications [8, 9, 50, 65].”
Stacked Hourglass Networks for Human Pose Estimation (2016, methodology): “Other attempts to improve pose estimation performance include iterative or multi-stage methods [2, 11], cascaded refinement of predictions [8], and leveraging of additional information such as depth or motion cues [4, 7].”
Highly accurate protein structure prediction with AlphaFold (2021, neutral): “The iterative refinement using the whole network (which we term ‘recycling’ and is related to approaches in computer vision 28,29 ) contributes markedly to accuracy with minor extra training time (see Supplementary Methods 1.8 for details).”
2D/3D Pose Estimation and Action Recognition Using Multitask Deep Learning (2018, methodology): “Our approach for human pose estimation is a regression method, similarly to [28, 47, 9].”
End-to-End Recovery of Human Shape and Pose (2017, methodology): “In [7, 31] the estimates are rendered to an image space to concatenate with the image input.”
FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks (2016, background): “This led to improved results in human pose estimation [17, 8] and semantic instance segmentation [22].”
Single Image 3D Interpreter Network (2016, background): “Recently, there have been several attempts to apply convolutional neural networks to human keypoint estimation [31,32,33,34], which all achieved signiﬁcant improvement.”
Iterative Instance Segmentation (2015, methodology): “IEF [6] uses a related approach on the task of human pose estimation by directly refining the prediction rather than approximating message passing or oracle output in each iteration.”
Structured Prediction Energy Networks (2015, methodology): “(Carreira et al., 2015) propose an iterative structured prediction method for human pose estimation, where predictions are constructed incrementally as yt+1 = yt + ∆(x, yt).”
Recurrent Human Pose Estimation (2016, methodology): “However, unlike [9] our model is trained end-to-end and does not”
Compositional Human Pose Regression (2017, methodology): “In this experiment, it is integrated into the Iterative Error Feedback method (IEF) [6], which is the only regression based method in the family.”
Deep High-Resolution Representation Learning for Human Pose Estimation (2019, methodology): “There are two mainstream methods: regressing the position of keypoints [64, 7], and estimating keypoint heatmaps [13, 14, 75] followed by choosing the locations with the highest heat values as the keypoints.”
Pose Guided Person Image Generation (2017, neutral): “and appearance details respectively similar to [35, 9, 3, 19].”