Chained Predictions Using Convolutional Neural Networks

By Georgia Gkioxari, Alexander Toshev, Navdeep Jaitly. 2016

199 citations recorded by the source corpus.

In this work, we present an adaptation of the sequence-to-sequence model for structured vision tasks. In this model, the output variables for a given input are predicted sequentially using neural networks. The prediction for each output variable depends not only on the input but also on the previously predicted output variables. The model is applied to spatial localization tasks and uses convolutional neural networks (CNNs) for processing input images and a multi-scale deconvolutional architecture for making spatial predictions at each step. We explore the impact of weight sharing with a recurrent connection matrix between consecutive predictions, and compare it to a formulation where these weights are not tied. Untied weights are particularly suited for problems with a fixed sized structure, where different classes of output are predicted at different steps. We show that chain models achieve top performing results on human pose estimation from images and videos.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Towards Accurate Multi-person Pose Estimation in the Wild (2017, methodology): “Recently, there has been signiﬁcant progress on this problem, mostly by leveraging deep Convolutional Neural Networks (CNNs) trained on large labeled datasets [43, 26, 42, 10, 31, 2, 7, 6, 20, 24, 8].”
2D/3D Pose Estimation and Action Recognition Using Multitask Deep Learning (2018, methodology): “Detection based methods handle pose estimation as a heat map prediction problem, where each pixel in a heat map represents the detection score of a corresponding joint [7, 18].”
Deep High-Resolution Representation Learning for Human Pose Estimation (2019, background): “Nowadays, deep convolutional neural network provides dominant solutions [20, 34, 60, 41, 42, 47, 56, 16].”
LCR-Net++: Multi-Person 2D and 3D Pose Detection in Natural Images (2018, unknown): “[73] 96.”
Learning Feature Pyramids for Human Pose Estimation (2017, unknown): “[20] 96.”
Integral Human Pose Regression (2017, methodology): “Most recent successful methods adopt a common multistage architecture [43, 5, 29, 4, 15, 19, 12, 10, 8, 32].”
Next-Step Conditioned Deep Convolutional Neural Networks Improve Protein Secondary Structure Prediction (2017, methodology): “This technique has led to substantial improvements in the performance of sequence-to-sequence models on complex tasks such as machine translation and pose prediction (Sutskever, Vinyals, & Le, 2014; Bengio et al ., 2015; Gkioxari, Toshev, & Jaitly, 2016).”
UViM: A Unified Modeling Approach for Vision with Learned Guiding Codes (2022, methodology): “The idea of modeling a complex structured target by recurrent “autoregressive” invocations of a model was used in a customized implementations for visual relationship prediction [24] and human pose estimation [12].”
Multi-context Attention for Human Pose Estimation (2017, unknown): “[16] 96.”
Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression (2021, methodology): “Early CNN approaches [62, 2, 8] directly predict the keypoint positions for single-person pose estimation, which is later surpassed by the heatmap estimation based methods [5, 20, 13, 37, 1].”
Human Pose Regression by Combining Indirect Part Detection and Contextual Information (2017, methodology): “[17] presented a structured prediction method, where the prediction of each joint depends on the intermediate feature maps and the distribution probability of the previously pre-”
Sequence-to-Sequence Models Can Directly Translate Foreign Speech (2017, methodology): “Subsequently, the model has been adapted and applied to various tasks such as image captioning [3, 4], pose prediction [5], and syntactic parsing [6].”
Pose for Action - Action for Pose (2016, methodology): “amp; Ramanan [38] 57.9 51.3 30.1 21.4 52.6 49.7 46.2 44.2 — Park & Ramanan [16] 62.8 52.0 32.3 23.3 53.3 50.2 43.0 45.3 — Nie et al. [15] 64.2 55.4 33.8 24.4 56.4 54.1 48.0 48.0 — Gkioxari et al. [10] 95.6 93.8 90.4 90.7 91.8 90.8 91.5 91.8 — TABLE IV: Comparison with the state-of-the-art in ter”
Monocular human pose estimation: A survey of deep learning-based methods (2020, methodology): “Gkioxari et al. (2016) proposed a convolutional Recurrent Neural Network to output joint location one by one following a chain model.”
Knowledge-Guided Deep Fractal Neural Networks for Human Pose Estimation (2017, unknown): “, ECCV’16 [68] 96.”
Learning to Fuse 2D and 3D Image Cues for Monocular Body Pose Estimation (2016, background): “With the growing availability of large datasets and the advent of Deep Learning, the emphasis has shifted towards using discriminative 2D pose regressors [11, 13, 15, 16, 24, 27, 32, 43, 47, 48, 67, 71, 72] to extract the 2D pose and infer a 3D one from it [9, 19, 73, 77].”
Monocular 3D Human Pose Estimation Using Transfer Learning and Improved CNN Supervision (2016, methodology): “In 2D joint detection and pose estimation from RGB, data-driven approaches using Convolutional Neural Networks (CNNs) have shown impressive results [18, 82, 78, 79, 28, 51, 14, 9, 47, 42, 15, 25, 27], outperforming previous hand crafted and model-based methods by a large margin [1, 5, 22].”