A Simple, Fast and Highly-Accurate Algorithm to Recover 3D Shape from 2D Landmarks on a Single Image

By Ruiqi Zhao, Yan Wang, Aleix M. Martinez. 2016

55 citations recorded by the source corpus.

Three-dimensional shape reconstruction of 2D landmark points on a single image is a hallmark of human vision, but is a task that has been proven difficult for computer vision algorithms. We define a feed-forward deep neural network algorithm that can reconstruct 3D shapes from 2D landmark points almost perfectly (i.e., with extremely small reconstruction errors), even when these 2D landmarks are from a single image. Our experimental results show an improvement of up to two-fold over state-of-the-art computer vision algorithms; 3D shape reconstruction error (measured as the Procrustes distance between the reconstructed shape and the ground-truth) of human faces is <inline-formula><tex-math notation="LaTeX">$<.004$</tex-math><alternatives> <inline-graphic xlink:href="martinez-ieq1-2772922.gif"/></alternatives></inline-formula>, cars is .0022, human bodies is .022, and highly-deformable flags is .0004. Our algorithm was also a top performer at the 2016 3D Face Alignment in the Wild Challenge competition (done in conjunction with the European Conference on Computer Vision, ECCV) that required the reconstruction of 3D face shape from a single image. The derived algorithm can be trained in a couple hours and testing runs at more than 1,000 frames/s on an i7 desktop. We also present an innovative data augmentation approach that allows us to train the system efficiently with small number of samples. And the system is robust to noise (e.g., imprecise landmark points) and missing data (e.g., occluded or undetected landmark points).

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Feature Boosting Network For 3D Pose Estimation (2019, neutral): “[41] 43.6% 56.8% 70.1% ICPPSO [40] 52.0% 64.5% 71.7% CHPR [40] 56.6% 71.7% 82.2% Zimmermann et al .”
Holistic++ Scene Understanding: Single-View 3D Holistic Scene Parsing and Human Pose Estimation With Human-Object Interaction and Physical Commonsense (2019, background): “estimation and then reconstruct 3D pose from the estimated 2D joints [53, 27, 32, 48, 6, 44].”
Lifting from the Deep: Convolutional 3D Pose Estimation from a Single Image (2017, background): “[47] achieved state-of-the-art results by training a simple neural network to recover 3D pose from known 2D joint positions.”
Sim2real transfer learning for 3D human pose estimation: motion to the rescue (2019, methodology): “One straightforward approach is to ‘lift’ 2D poses into 3D, using either dictionaries or direct regression [3, 44, 47, 64, 79, 85, 88, 93].”
Learning to Estimate 3D Hand Pose from Single RGB Images (2017, methodology): “Figure 9: Results for our complete system on S-val compared to approaches from [25] and [26].”