Sim2real transfer learning for 3D human pose estimation: motion to the rescue

By Carl Doersch, Andrew Zisserman. 2019

170 citations recorded by the source corpus.

Synthetic visual data can provide practicically infinite diversity and rich labels, while avoiding ethical issues with privacy and bias. However, for many tasks, current models trained on synthetic data generalize poorly to real data. The task of 3D human pose estimation is a particularly interesting example of this sim2real problem, because learning-based approaches perform reasonably well given real training data, yet labeled 3D poses are extremely difficult to obtain in the wild, limiting scalability. In this paper, we show that standard neural-network approaches, which perform poorly when trained on synthetic RGB images, can perform well when the data is pre-processed to extract cues about the person’s motion, notably as optical flow and the motion of 2D keypoints. Therefore, our results suggest that motion can be a simple way to bridge a sim2real gap when video is available. We evaluate on the 3D Poses in the Wild dataset, the most challenging modern benchmark for 3D pose estimation, where we show full 3D mesh recovery that is on par with state-of-the-art methods trained on real 3D sequences, despite training only on synthetic humans from the SURREAL dataset.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Perceiver IO: A General Architecture for Structured Inputs & Outputs (2021, background): “This has many broader applications, such as navigation and visual odometry in robots [11], estimation of 3D geometry [66], and even to aid transfer of more complex, learned inference such as 3D human pose estimation from synthetic to real images [23].”
HuMoR: 3D Human Motion Model for Robust Pose Estimation (2021, background): “This has led to developments in predicting motion (pose sequences) and shape directly from RGB video [42, 105, 81, 87, 22].”
Deep Learning-based Human Pose Estimation: A Survey (2020, methodology): “, [157] [158] [159] [160] [161] [162] [163] [164] [165] [166], because it is compatible with existing rendering engines.”
TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement (2023, unknown): “One interesting exception uses sim2real based on motion [14], motivating better surface tracking.”
VIBE: Video Inference for Human Body Pose and Shape Estimation (2019, methodology): “[16] 74.”
PyMAF: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loop (2021, background): “These deep regressors take 2D evidences as input and learn model priors implicitly in a data-driven manner under different types of supervision signals [53, 22, 41, 43, 11, 61, 28] during the learning procedure.”