Self-supervised Learning of Motion Capture

By Hsiao-Yu Fish Tung, Hsiao-Wei Tung, Ersin Yumer, Katerina Fragkiadaki. 2017

332 citations recorded by the source corpus.

Current state-of-the-art solutions for motion capture from a single camera are optimization driven: they optimize the parameters of a 3D human model so that its re-projection matches measurements in the video (e.g. person segmentation, optical flow, keypoint detections etc.). Optimization models are susceptible to local minima. This has been the bottleneck that forced using clean green-screen like backgrounds at capture time, manual initialization, or switching to multiple cameras as input resource. In this work, we propose a learning based motion capture model for single camera input. Instead of optimizing mesh and skeleton parameters directly, our model optimizes neural network weights that predict 3D shape and skeleton configurations given a monocular RGB video. Our model is trained using a combination of strong supervision from synthetic data, and self-supervision from differentiable rendering of (a) skeletal keypoints, (b) dense 3D mesh motion, and (c) human-background segmentation, in an end-to-end framework. Empirically we show our model combines the best of both worlds of supervised learning and test-time optimization: supervised learning initializes the model parameters in the right regime, ensuring good pose and surface initialization at test time, without manual effort. Self-supervision by back-propagating through differentiable rendering allows (unsupervised) adaptation of the model to the test data, and offers much tighter fit than a pretrained fixed model. We show that the proposed model improves with experience and converges to low-error solutions where previous optimization methods fail.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

End-to-End Recovery of Human Shape and Pose (2017, methodology): “er ﬁxed to minimize the silhouette reprojection loss. However, the reliance on silhouettes limits their approach to frontal images and images of humans without any occlusion. Concurrently Tung et al. [46] predict SMPL parameters from an image and a set of 2D joint heatmaps. The model is pretrained o”
Tracking Emerges by Colorizing Videos (2018, background): “A related but diﬀerent line of work explores how to learn geometric properties or cycle consistencies with self-supervision, for example for motion capture or correspondence [20–24].”
Learning to Reconstruct 3D Human Pose and Shape via Model-Fitting in the Loop (2019, methodology): “This information can be used as input [37], intermediate representation [24, 27], or as supervision, by enforcing different reprojection losses [15, 24, 27, 34, 37].”
Combining Implicit Function Learning and Parametric Models for 3D Human Reconstruction (2020, background): “y slow, and are susceptible to local-minima. In contrast, deep learning based models predict body model parameters in a feed-forward pass [16,43,39] and use bottom-up 2D features for self-supervision [21,23,24,32,50,60] during training. These approaches are limited by the shape space of SMPL, can no”
NASA: Neural Articulated Shape Approximation (2019, methodology): “Although parametric models of human body such as SMPL [33] have been integrated into neural network frameworks for self-supervision [27,38,40,55], these approaches depend heavily on polygonal mesh representations.”
Learning 3D Human Dynamics From Video (2018, methodology): “There are some recent works that output smooth 3D pose and shape: [51] predicts SMPL parameters from two video frames by using optical flow, silhouettes, and keypoints in a self-supervised manner.”
PyMAF: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loop (2021, methodology): “To mitigate the learning difficulty of the regressor, different network architectures have also been designed to leverage proxy representations such as silhouette [42, 54], 2D/3D joints [53, 42, 37], segmentation [40, 45] and dense correspondences [59, 66].”
VIBE: Video Inference for Human Body Pose and Shape Estimation (2019, methodology): “Due to the lack of in-the-wild 3D ground-truth labels, these methods use weak supervision signals obtained from a 2D keypoint re-projection loss [29, 60, 62], use body/part segmentation as an intermediate representation [48, 51], or employ a human in the loop [39].”
PARE: Part Attention Regressor for 3D Human Body Estimation (2021, methodology): “In order to deal with the lack of in-the-wild 3D ground-truth, methods use a 2D keypoint re-projection loss as weak supervision [24, 50, 51], use intermediate 2D representations, e.”
3D Hand Shape and Pose From Images in the Wild (2019, background): “Model-based decoders Several works propose to combine deep convolutional encoders with generative models as decoders for human face [56, 55] and body [17, 60] 3D reconstruction.”
Convolutional Mesh Regression for Single-Image Human Shape Reconstruction (2019, methodology): “[43] present a self-supervised approach for the same problem, while Tan et al.”
DeepHuman: 3D Human Reconstruction From a Single Image (2019, methodology): “For example, by fitting the SCAPE or SMPL model to the detected 2D skeleton and other shape cues of an image [5, 28], or by regressing [27, 49, 51] the SMPL model using CNNs, the shape and pose parameters can be automatically obtained from a single image.”
Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion (2020, background): “Meshes for humans: Since the introduction of the (mesh-based) SMPL human model [41] there have been a growing number of papers leveraging it to reconstructing shape and pose from point clouds, depth data and images [28, 32, 33, 47, 69, 79].”
Ego-Pose Estimation and Forecasting As Real-Time PD Control (2019, unknown): “Deep learning based approaches [70, 35, 32, 57] have also succeeded in directly regressing images to 3D joint locations with the help of large-scale MoCap datasets [15].”
Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wild (2020, background): “Another line of work develops a learning-based framework, using a feed-forward model to directly predict the parameters of the body model from a single image [61,53,30,47,63,44].”
Deep Learning-based Human Pose Estimation: A Survey (2020, unknown): “, [157] [158] [159] [160] [161] [162] [163] [164] [165] [166], because it is compatible with existing rendering engines.”
Discovery of Latent 3D Keypoints via End-to-end Geometric Reasoning (2018, methodology): “Besides keypoints, self-supervision based on geometric and motion reasoning has been used to predict other forms of output, such as 3D shape represented as blendshape coefficients for human motion capture [57].”
A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose (2021, background): “It also enables optimization within the bounds of the learned prior [14, 19, 28] and weak-supervision when integrated in a differentiable form [30] into neural training processes [5, 23, 26, 45, 48, 68].”