Learning Pose Grammar to Encode Human Body Configuration for 3D Pose Estimation

By Haoshu Fang, Yuanlu Xu, Wenguan Wang, Xiaobai Liu, Song-Chun Zhu. AAAI Conference on Artificial Intelligence, 2017

391 citations recorded by the source corpus.

In this paper, we propose a pose grammar to tackle the problem of 3D human pose estimation. Our model directly takes 2D pose as input and learns a generalized 2D-3D mapping function. The proposed model consists of a base network which efficiently captures pose-aligned features and a hierarchy of Bi-directional RNNs (BRNN) on the top to explicitly incorporate a set of knowledge regarding human body configuration (i.e., kinematics, symmetry, motor coordination). The proposed model thus enforces high-level constraints over human poses. In learning, we develop a pose sample simulator to augment training samples in virtual camera views, which further improves our model generalizability. We validate our method on public 3D human pose benchmarks and propose a new evaluation protocol working on cross-view setting to verify the generalization capability of different methods. We empirically observe that most state-of-the-art methods encounter difficulty under such setting while our method can well handle such challenges.

View Paper: https://www.semanticscholar.org/paper/db466d02ec9f7f47ee11590d0feeb6330256c98d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Wenguan Wang
2. Jun Liu
3. Jitendra Malik
4. Luc Van Gool
5. Jianbing Shen
6. Pichao Wang
7. Jiefeng Li
8. Long Zhao

Citations in the corpus

• Compositional Human Pose Regression (2017, methodology): “2D joints are first estimated and then used to estimate the 3D pose via learning a 2D to 3D mapping (Moreno-Noguer, 2016; Martinez et al., 2017; Fang et al., 2018), fitting a 3D model via optimization (Zhou et al.”
• Integral Human Pose Regression (2017, unknown): “1) [21]∗ [14]∗ [51] [41] [8] [32] [54] [30] [26] [42] [17]”
• CrowdPose: Efficient Crowded Scenes Pose Estimation and a New Benchmark (2018, background): “Some works focus on 3D human pose estimation [33, 34, 31].”
• Camera Distance-Aware Top-Down Approach for 3D Multi-Person Pose Estimation From a Single RGB Image (2019, unknown): “[4, 5, 7, 26, 35, 49, 52] are based on the two-stage approach.”
• 3D Human Pose Estimation in Video With Temporal Convolutions and Semi-Supervised Training (2018, methodology): “Protocol 2 reports the error after alignment with the ground truth in translation, rotation, and scale (P-MPJPE) [34, 50, 10, 40, 58, 16].”
• Holistic++ Scene Understanding: Single-View 3D Holistic Scene Parsing and Human Pose Estimation With Human-Object Interaction and Physical Commonsense (2019, background): “Based on one single-view image, existing work either focuses only on 3D holistic scene understanding [16, 62, 3, 40] or 3D human pose estimation [53, 32, 9].”
• Monocular Total Capture: Posing Face, Body, and Hands in the Wild (2018, methodology): “Method Pavlakos [42] Zhou [73] Luo [31] Martinez [33] Fang [20] Yang [70] Pavlakos [41] Dabral [18] Sun [56] *Kanazawa [27] *Mehta [35] *Mehta [34] *Ours *Ours+”
• Exploiting Temporal Contexts With Strided Transformer for 3D Human Pose Estimation (2021, methodology): “The mean per joint position error (MPJPE) is the average Euclidean distance between the ground truth and predicted positions of the joints, which is referred to as protocol #1 in many works [41], [58].”
• Semantic Graph Convolutional Networks for 3D Human Pose Regression (2019, methodology): “We compared ours with three GCN-based methods: aGCN [68], GAT [60] and ST-GCN [67], and two state-of-the-art approaches: FC [34] and PG [13].”
• HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimation (2020, methodology): “Two-stage approaches first estimate 2D pose and then lift them to 3D joint locations by a learned dictionary of 3D skeleton [1, 55, 66, 58, 78, 79] or regression [48, 73, 42, 11, 34, 61].”
• Learning Human-Object Interactions by Graph Parsing Neural Networks (2018, background): “Thus pre-fixed graph structures, as adopted by most previous graphical or structured DNN models [11, 20, 22, 43], are not an optimal choice.”
• Reasoning Visual Dialogs With Structural and Partial Observations (2019, background): “Another stream is to build graphically structured neural networks to approximate the learning/inference process of graphical models [30, 55, 29, 14, 4, 17, 60, 8].”
• Exploiting Spatial-Temporal Relationships for 3D Pose Estimation via Graph Convolutional Networks (2019, methodology): “To incorporate spatial priors into the framework, Fang [12] developed a deep grammar network to explicitly encode the human body dependencies and relations.”
• ARCH: Animatable Reconstruction of Clothed Humans (2020, background): “However, they usually rely on representations that have limitations: (1) skeletons [11] are kinematic structures that are accurate to represent 3D poses, but do not carry body shape information.”
• Human Pose Regression with Residual Log-likelihood Estimation (2021, methodology): “Two-stage methods [32, 13, 47, 68, 60, 9, 30, 67] lift the 2D poses to 3D space by regression.”
• Self-Supervised Learning of 3D Human Pose Using Multi-View Geometry (2019, methodology): “Additionally, there are two-stage approaches which de-compose the 3D pose inference task into two independent stages: estimating 2D poses, and lifting them into 3D space [8, 24, 22, 11, 46, 8, 40, 23].”