Articulated mesh animation from multi-view silhouettes

By Daniel Vlasic, Ilya Baran, Wojciech Matusik, Jovan Popovic. ACM Transactions on Graphics, 2008

790 citations recorded by the source corpus.

Details in mesh animations are difficult to generate but they have great impact on visual quality. In this work, we demonstrate a practical software system for capturing such details from multi-view video recordings. Given a stream of synchronized video images that record a human performance from multiple viewpoints and an articulated template of the performer, our system captures the motion of both the skeleton and the shape. The output mesh animation is enhanced with the details observed in the image silhouettes. For example, a performance in casual loose-fitting clothes will generate mesh animations with flowing garment motions. We accomplish this with a fast pose tracking method followed by nonrigid deformation of the template to fit the silhouettes. The entire process takes less than sixteen seconds per frame and requires no markers or texture cues. Captured meshes are in full correspondence making them readily usable for editing operations including texturing, deformation transfer, and deformation model learning.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/af8990bc767da54a3bfb4c01e04f9bd1ed09396d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Control and Systems Engineering

Citations in the corpus

Robust single-view geometry and motion reconstruction (2009, methodology): “More recently, several papers avoid the use of markers to reproduce complex animations of human performances and cloth deformations from multi-view video [Bradley et al. 2008; de Aguiar et al. 2008; Vlasic et al. 2008].”
HumanEva: Synchronized Video and Motion Capture Dataset and Baseline Algorithm for Evaluation of Articulated Human Motion (2010, background): “Current research in non-marker-based methods for capturing human motion (Vlasic et al. 2008) may prove to be viable alternatives in a few years.”
Dynamic FAUST: Registering Human Bodies in Motion (2017, background): “For example, there are many non-rigid tracking methods, either model-based [3, 7, 22, 25, 27, 34, 44, 49] or model-free [2, 16, 21, 24, 37, 50, 54].”
Panoptic Studio: A Massively Multiview System for Social Interaction Capture (2016, methodology): “Other methods explicitly tackle the markerless motion capture by producing 3D skeletal structures over time similar to the marker-based counterparts [24], [25], [26], [27], [28], [29], [30], [31], [32], [33], [34], [35], [36].”
NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction (2022, background): “A large body of work [6, 57, 2, 9, 64, 16] in multi-view 4D reconstruction utilizes a precomputed deformable model, which is then fit to the multi-view images.”
The wave kernel signature: A quantum mechanical approach to shape analysis (2011, methodology): “In the following we will present both qualitative and quantitative evidence showing that the proposed Wave Kernel Signature is superior to the traditional Heat Kernel Signature .”
Panoptic Studio: A Massively Multiview System for Social Motion Capture (2015, background): “To obtain greater detail in the 3D reconstruction, de Aguiar et al. [12], Vlasic et al. [39], and Furukawa and Ponce [16] deformed pre-deﬁned templates of ﬁxed topology to recover details that were subsampled or occluded in the set of views at a time instant.”
PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization (2019, background): “Early attempts are based on visual hulls [37, 61, 15, 14] which uses silhouettes from multiple views to carve out the visible areas of a capture volume.”
Video Based Reconstruction of 3D People Models (2018, neutral): “In [23, 69] they pre-scan a template and insert a skeleton and in [78] they use a skeleton to regularize dynamic fusion.”
Motion capture using joint skeleton tracking and surface estimation (2009, methodology): “Our approach is most similar to the work of Vlasic et al. [28] where a two-pass approach has been proposed.”
Dense Human Body Correspondences Using Convolutional Networks (2015, methodology): “Note that the Yobi3D dataset covers the shape variability in local geometry, while the SCAPE and the MIT datasets cover the variability in pose.”
ClothCap (2017, methodology): “Vlasic et al. (2008) propose a multi-pass approach that involves fitting a skeleton to visual hulls, deforming the template according to the skeleton, and finally deforming the template to fit image silhouettes.”
Capturing the human figure through a wall (2015, background): “On the other hand, past markerless methods use cameras and infrared-based techniques – including Kinect, multi-view cameras, moving cameras, and time-of-ﬂight cameras – and require a direct line-of-sight from the sensor to the person’s body, e.g., [Shotton et al. 2013; Ganapathi et al. 2010; Hasler ”