Humans in 4D: Reconstructing and Tracking Humans with Transformers

By Shubham Goel, Georgios Pavlakos, Jathushan Rajasegaran, Angjoo Kanazawa, Jitendra Malik. IEEE International Conference on Computer Vision, 2023

444 citations recorded by the source corpus.

We present an approach to reconstruct humans and track them over time. At the core of our approach, we propose a fully "transformerized" version of a network for human mesh recovery. This network, HMR 2.0, advances the state of the art and shows the capability to analyze unusual poses that have in the past been difficult to reconstruct from single images. To analyze video, we use 3D reconstructions from HMR 2.0 as input to a tracking system that operates in 3D. This enables us to deal with multiple people and maintain identities through occlusion events. Our complete approach, 4DHumans, achieves state-of-the-art results for tracking people from monocular video. Furthermore, we demonstrate the effectiveness of HMR 2.0 on the downstream task of action recognition, achieving significant improvements over previous pose-based action recognition approaches. Our code and models are available on the project website: https://shubham-goel.github.io/4dhumans/.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/d865fe9a755473301f0560c1d3b102035a14848a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Champ: Controllable and Consistent Human Image Animation with 3D Parametric Guidance (2024, methodology): “Given a reference human image I ref and a sequence of driving motion video frames I 1: N , where N denotes the total number of frames, we obtain the 3D human parametric SMPL model, H ref and H 1: N m , respectively, utilizing an existing framework known as 4D-Humans [8].”
Reconstructing Hands in 3D with Transformers (2023, methodology): “HaMeR adopts a simple architecture with a fully transformer-based design (Figure 2, bottom), similar to [17].”
LRM: Large Reconstruction Model for Single Image to 3D (2023, unknown): “…to model the image-to-3D data distribution (Choy et al., 2016; Yan et al., 2016; Dai et al., 2017; Xu et al., 2019; Wu et al., 2020; M¨uller et al., 2022; Sajjadi et al., 2022; Goel et al., 2023), where a compact latent code is trained to carry the texture, geometry, and pose details of the target.”