Dense Human Body Correspondences Using Convolutional Networks

By Lingyu Wei, Qi-Xing Huang, Duygu Ceylan, Etienne Vouga, Hao Li. 2015

211 citations recorded by the source corpus.

We propose a deep learning approach for finding dense correspondences between 3D scans of people. Our method requires only partial geometric information in the form of two depth maps or partial reconstructed surfaces, works for humans in arbitrary poses and wearing any clothing, does not require the two people to be scanned from similar view-points, and runs in real time. We use a deep convolutional neural network to train a feature descriptor on depth map pixels, but crucially, rather than training the network to solve the shape correspondence problem directly, we train it to solve a body region classification problem, modified to increase the smoothness of the learned descriptors near region boundaries. This approach ensures that nearby points on the human body are nearby in feature space, and vice versa, rendering the feature descriptor suitable for computing dense correspondences between the scans. We validate our method on real and synthetic data for both clothed and unclothed humans, and show that our correspondences are more robust than is possible with state-of-the-art unsupervised methods, and more accurate than those found using methods that require full watertight 3D geometry.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

Geometric Deep Learning on Graphs and Manifolds Using Mixture Model CNNs (2016, background): “In deformable 3D shape analysis applications, the key advantage of our approach is that it is intrinsic and thus deformation-invariant by construction, as opposed to Euclidean models [36, 43, 41, 29] that in general require significantly higher complexity and huge training sets to learn the deformat”
DensePose: Dense Human Pose Estimation in the Wild (2018, background): “The task of establishing dense correspondences from an image to a surface-based model has been addressed mostly in the setting where a depth sensor is available [43, 34, 46].”
Geometric Deep Learning: Going beyond Euclidean data (2016, background): “as range images [98], [99] or rasterized volumes [100], [101].”
Dynamic Graph CNN for Learning on Point Clouds (2018, methodology): “oes not have an underlying grid, requiring new building blocks replacing convolution and pooling or adaptation to a grid structure. As a simple way to overcome this issue, view-based [Su et al. 2015; Wei et al. 2016] and volumetric representations [Klokov and Lempitsky 2017; Maturana and Scherer 201”
PCPNet Learning Local Shape Properties from Raw Point Clouds (2017, methodology): “However, several methods have also been proposed for shape segmentation [GZC15,MGA ∗ 17] and even shape correspondence [MBBV15,WHC ∗ 16,BMRB16], among many others.”
Learning shape correspondence with anisotropic convolutional neural networks (2016, methodology): “[26] used view-based representation to find correspondence between non-rigid shapes.”
VConv-DAE: Deep Volumetric Shape Learning Without Object Labels (2016, methodology): “ uses 3D-2D(multi-view images) correspondence to improve shape completion (repairing) results [11,12] propose intrinsic CNN [13,14] for 3D data or learn correspondence between two surfaces(depth map) [15]. Our work is mainly inspired by Shapenet in the functionality but diers in methodology. In part”
Deep Functional Maps: Structured Prediction for Dense Shape Correspondence (2017, methodology): “[41] employed a classical (extrinsic) CNN architecture trained on huge training sets for learning invariance to pose changes and clothing.”
Deformable Shape Completion with Graph Convolutional Autoencoders (2017, methodology): “Several works avoid this problem by using a Euclidean representation of 3D shapes, such as rendering a collection of 2D views [52, 59], volumetric representations [61], or point cloud [42, 43].”
SurfNet: Generating 3D Shape Surfaces Using Deep Residual Networks (2017, methodology): “Methods for non-rigid shape correspondence such as [6, 19, 34] can be used to develop dense one-to-one correspondence between mesh models when the correspondence information is unavail-Figure able.”
Image Matching from Handcrafted to Deep Features: A Survey (2020, methodology): “Wei et al. (2016) trained a deep CNN on the depth map representation of shapes to find correspondences.”
Geometric deep learning (2016, background): “The text in the course materials is primarily based on previously published work including [71, 56, 86, 100, 10, 59, 97, 12, 11] among several others.”
Total Capture: 3D Human Pose Estimation Fusing Video and Inertial Sensors (2017, background): “The challenge of estimating 3D human pose from MVV is currently less explored, although initial work by Trumble [32] used MVV with a simple 2D convolutional neural network (convnet), and Wei [36] performed related work aligning pairs of 3D human pose.”
Fast end-to-end learning on protein surfaces (2020, background): “based on 2D views [44], volumetric [38], space partitioning [35, 42, 39] and implicit representations [14]), geometric deep learning seeks to develop a non-Euclidean analogy of filtering and pooling operations.”
Learning Representations and Generative Models for 3D Point Clouds (2017, background): “Recently, deep learning architectures for view-based projections (Su et al., 2015; Wei et al., 2016; Kalogerakis et al., 2016), volumetric grids (Qi et al.”
Dynamic FAUST: Registering Human Bodies in Motion (2017, methodology): “From this dataset, researchers have used both real scans (for benchmarking registration techniques [20, 51, 55]) and aligned templates (e.”
Combining Implicit Function Learning and Parametric Models for 3D Human Reconstruction (2020, methodology): “ences on a depth map with a random forest and t a 3D model to them [38,37,48]. To predict correspondences from point clouds using CNNs, depth maps can be generated where convolutions can be performed [52]. Our approach diers critically in that i) we do not require generating multiple depth maps, 2) ”
PointCleanNet: Learning to Denoise and Remove Outliers from Dense Point Clouds (2019, methodology): “Inspired by the recent successes of applying deep learning techniques for the analysis and processing of geometric data, including [MBBV15,BBL ∗ 17,WHC ∗ 16] among many others, and especially the seminal works designed for learning directly on point clouds [QSMG17, WSL ∗ 18], in this paper, we…”