Real-time Facial Surface Geometry from Monocular Video on Mobile GPUs

By Yury Kartynnik, Artsiom Ablavatski, Ivan Grishchenko, Matthias Grundmann. arXiv.org, 2019

334 citations recorded by the source corpus.

We present an end-to-end neural network-based model for inferring an approximate 3D mesh representation of a human face from single camera input for AR applications. The relatively dense mesh model of 468 vertices is well-suited for face-based AR effects. The proposed model demonstrates super-realtime inference speed on mobile GPUs (100-1000+ FPS, depending on the device and model variant) and a high prediction quality that is comparable to the variance in manual annotations of the same image.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Signal Processing

View Paper: https://www.semanticscholar.org/paper/75c7b100aaad9d12819e5e3802986c6f2e9da0b0

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

Artificial intelligence in the creative industries: a review (2020, methodology): “ch has created software for pose animation that turns a human pose into a cartoon animation in real time30. This is based on PoseNet (estimating pose position31) and FaceMesh (capturing face movement [18]) as shown in Fig. 5 (a). Adobe has also created Character Animator software32 oering lip synchr”
MediaPipe Hands: On-device Real-time Hand Tracking (2020, methodology): “To recover from tracking failure, we developed another output of the model similar to [8] for producing the probability of the event that a reasonably aligned hand is indeed present in the provided crop.”
BlazePose: On-device Real-time Body Pose tracking (2020, methodology): “As its 33 keypoint topology is consistent with BlazeFace[2] and BlazePalm[3], it can be a backbone for subsequent hand pose[3] and facial geometry estimation[6] models.”
Portrait shadow manipulation (2020, methodology): “ by warping image content near each vertex of a mesh to the location of its corresponding mirrored vertex. See Figure 4 for a visualization. Given an image I, we use the landmark detection system of (Kartynnik et al. 2019) to produce a model of facial geometry consisting of 468 2D vertices (Figure 4”