Reconstructing Hands in 3D with Transformers

By Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David F. Fouhey, Jitendra Malik. Computer Vision and Pattern Recognition, 2023

381 citations recorded by the source corpus.

We present an approach that can reconstruct hands in 3D from monocular input. Our approach for Hand Mesh Recovery, HaMeR, follows a fully transformer-based architecture and can analyze hands with significantly increased accuracy and robustness compared to previous work. The key to HaMeR's success lies in scaling up both the data used for training and the capacity of the deep network for hand reconstruction. For training data, we combine multiple datasets that contain 2D or 3D hand annotations. For the deep model, we use a large scale Vision Transformer architecture. Our final model consistently outperforms the previous baselines on popular 3D hand pose benchmarks. To further evaluate the effect of our design in non-controlled settings, we annotate existing in-the-wild datasets with 2D hand keypoint annotations. On this newly collected dataset of annotations, HInt, we demonstrate significant improvements over existing baselines. We will make our code, data and models publicly available upon publication. We make our code, data and models available on the project website: https://geopavlakos.github.io/hamer/. “It is because of his being armed with hands that man is the most intelligent animal.” Anaxagoras

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, Geology

View Paper: https://www.semanticscholar.org/paper/327a546280368245b551183279b41023393334d4

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Zipeng Fu
2. Qingqing Zhao
3. Gordon Wetzstein
4. Qi Wu
5. Chelsea Finn
6. Sergey Levine
7. Luc Van Gool
8. Dacheng Tao

In Control and Systems Engineering

1. Zipeng Fu
2. Chelsea Finn
3. Qingqing Zhao
4. Gordon Wetzstein
5. Qi Wu
6. Sergey Levine
7. Pieter Abbeel
8. Suraj Nair

Citations in the corpus

• HumanPlus: Humanoid Shadowing and Imitation from Humans (2024, background): “Using state-of-the-art human body and hand pose estimation algorithms [58, 81], we can estimate real-time human motion and retarget it to humanoid motion, which is passed as input to the low-level policy.”