VIBE: Video Inference for Human Body Pose and Shape Estimation

By Muhammed Kocabas, Nikos Athanasiou, Michael J. Black. Computer Vision and Pattern Recognition, 2019

1,167 citations recorded by the source corpus.

Human motion is fundamental to understanding behavior. Despite progress on single-image 3D pose and shape estimation, existing video-based state-of-the-art methods fail to produce accurate and natural motion sequences due to a lack of ground-truth 3D motion data for training. To address this problem, we propose "Video Inference for Body Pose and Shape Estimation'' (VIBE), which makes use of an existing large-scale motion capture dataset (AMASS) together with unpaired, in-the-wild, 2D keypoint annotations. Our key novelty is an adversarial learning framework that leverages AMASS to discriminate between real human motions and those produced by our temporal pose and shape regression networks. We define a novel temporal network architecture with a self-attention mechanism and show that adversarial training, at the sequence level, produces kinematically plausible motion sequences without in-the-wild ground-truth 3D labels. We perform extensive experimentation to analyze the importance of motion and demonstrate the effectiveness of VIBE on challenging 3D pose estimation datasets, achieving state-of-the-art performance. Code and pretrained models are available at https://github.com/mkocabas/VIBE

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/decf55f42e15b3ce56c6bd14eb14906cec29b7ab

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting (2021, background): “Historically, RNNs ( e.g ., LSTMs [17], GRUs [7]) have achieved remarkable success in sequence modeling, with applications to speech recognition [52, 35], image captioning [53], machine translation [32], human pose estimation [56, 24], etc .”
Human Motion Diffusion Model (2022, background): “A large volume of motion generation research is devoted to learning using geometric losses (Kocabas et al., 2020; Harvey et al., 2020; Aberman et al., 2020).”
PaMIR: Parametric Model-Conditioned Implicit Representation for Image-Based Human Reconstruction (2020, methodology): “MPL model given a single image is a fundamentally ill-posed problem. Recent studies have explored to learn from largescale datasets to address this challenge [11], [12], [13], [14], [15], [37], [38], [39], [40]. In this work we use GCMR [13], one of the state-of-the-art networks as the backbone for ”
NeuMan: Neural Human Radiance Field from a Single Video (2022, background): “This is, in fact, a non-trivial problem, as human body pose estimators [13,42,18] operate in their own camera systems with often near-orthographic camera models.”
BEHAVE: Dataset and Method for Tracking Human Object Interactions (2022, neutral): “Human reconstruction and performance capture Per-ceiving humans from monocular RGB data [12, 29, 31, 41, 43, 44, 58, 59, 64, 87] and under multiple views [37–40 , 62] Following the success of pixel-aligned implicit function learning [64, 65], recent methods can capture human performance from sparse…”
Executing your Commands via Motion Diffusion in Latent Space (2022, methodology): “To evaluate reconstruction error of VAEs, we use popular metrics in motion capture [31, 8, 70], MPJPE and PAM-”
CLIFF: Carrying Location Information in Full Frames into Human Pose and Shape Estimation (2022, neutral): “Like most tasks in computer vision, there are two approaches to do this: top-down [16,23,20,21] and bottom-up [49,56].”
A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose (2021, methodology): “We follow two widely adopted test protocols denoted as Protocol I [23, 25, 26] and Protocol II [44, 62], in which we evaluate 14/17-joint estimation error on every 5/64 frame of the test videos, respectively.”
Action2Motion: Conditioned Generation of 3D Human Motions (2020, methodology): “We thus re-annotate this dataset with the help of a recent tool [15].”
MotionDiffuse: Text-Driven Human Motion Generation With Diffusion Model (2022, methodology): “ACTOR [2] further uses pre-trained VIBE [83] to extract SMPL [84] sequences from the UESTC dataset and provides pretrained action recognition model for evaluation.”
Deep Learning-based Human Pose Estimation: A Survey (2020, methodology): “[99] included the large-scale motion capture dataset AMASS [153] for adversarial training of their SMPL-based method named VIBE (Video Inference for Body Pose and Shape Estimation).”
Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video (2020, background): “Most existing approaches thus limit themselves to a single object category, such as the human body [23, 91, 32] or face [12].”
Revisiting Skeleton-based Action Recognition (2021, neutral): “The 3D skeletons are either collected by sensors (NTU-60) or estimated with the state-of-the-art 3D pose estimator [16] (FineGYM).”
D3D-HOI: Dynamic 3D Human-Object Interactions from Videos (2021, neutral): “Most works in this area focus on separately re-constructing the human mesh [1, 10, 11, 15, 16, 27] or the individual 3D object [4, 18, 22, 31].”