Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wild
By Dominik Kulon, Riza Alp Güler, Iasonas Kokkinos, Michael M. Bronstein, Stefanos P. Zafeiriou. 2020
244 citations recorded by the source corpus.
We introduce a simple and effective network architecture for monocular 3D hand pose estimation consisting of an image encoder followed by a mesh convolutional decoder that is trained through a direct 3D hand mesh reconstruction loss. We train our network by gathering a large-scale dataset of hand action in YouTube videos and use it as a source of weak supervision. Our weakly-supervised mesh convolutions-based system largely outperforms state-of-the-art methods, even halving the errors on the in the wild benchmark. The dataset and additional resources are available at https://arielai.com/mesh_hands.
Subfields: Computer Vision and Pattern Recognition, Human-Computer Interaction
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Jitendra Malik
- Luc Van Gool
- Cordelia Schmid
- Trevor Darrell
- Andrew Zisserman
- Xiaogang Wang
- Bernt Schiele
- Dacheng Tao
In Human-Computer Interaction
- Michael J. Black
- Jitendra Malik
- Angjoo Kanazawa
- Ilija Radosavovic
- Gul Varol
- Cordelia Schmid
- Georgios Pavlakos
- Ivan Laptev
Citations in the corpus
- Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications (2022, background): “One highly related direction is to model and estimate 3D hand joints [5,44,45,53,60,63,66,72,78,79,80,85,3] and mesh [26,27,51,55,75,83,46], object pose [14,28,29,35,43,57,64,71,76,81], or both [6,17,18,19,69].”
- Towards Unconstrained Joint Hand-Object Reconstruction From RGB Videos (2021, background): “3D pose estimation for hands [22, 31, 45, 56, 67] and objects [15, 26, 36, 39, 43, 49, 63, 64] have often been tackled in isolation [38].”
- DexMV: Imitation Learning for Dexterous Manipulation from Human Videos (2021, background): “One line of work focus on object pose estimation [36, 56, 85, 79, 50, 30, 27] and hand pose estimation [87, 31, 76, 23, 5, 11, 26, 38, 42].”
- Reconstructing Hands in 3D with Transformers (2023, background): “Fol-lowup work [10, 16, 31, 39] follows a non-parametric approach and directly regresses the vertices of the MANO mesh.”
- Reconstructing Hand-Object Interactions in the Wild (2020, methodology): “To tackle this setting, we employ a multi-step optimization procedure whose components build on prior work [24, 54, 19, 66, 56].”
- Grasping Field: Learning Implicit Representations for Human Grasps (2020, methodology): “Using the template derived from MANO, [56] show that it is also possible to regress hand meshes directly using mesh convolution.”