Leveraging Photometric Consistency Over Time for Sparsely Supervised Hand-Object Reconstruction

By Yana Hasson, Bugra Tekin, Federica Bogo, Ivan Laptev, Marc Pollefeys, Cordelia Schmid. 2020

226 citations recorded by the source corpus.

Modeling hand-object manipulations is essential for understanding how humans interact with their environment. While of practical importance, estimating the pose of hands and objects during interactions is challenging due to the large mutual occlusions that occur during manipulation. Recent efforts have been directed towards fully-supervised methods that require large amounts of labeled training samples. Collecting 3D ground-truth data for hand-object interactions, however, is costly, tedious, and error-prone. To overcome this challenge we present a method to leverage photometric consistency across time when annotations are only available for a sparse subset of frames in a video. Our model is trained end-to-end on color images to jointly reconstruct hands and objects in 3D by inferring their poses. Given our estimated reconstructions, we differentiably render the optical flow between pairs of adjacent images and use it within the network to warp one frame to another. We then apply a self-supervised photometric loss that relies on the visual consistency between nearby images. We achieve state-of-the-art results on 3D hand-object reconstruction benchmarks and demonstrate that our approach allows us to improve the pose estimation accuracy by leveraging information from neighboring frames in low-data regimes.

Subfields: Computer Vision and Pattern Recognition, 3D Vision & Pose, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

In Human-Computer Interaction

Citations in the corpus

Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications (2022, background): “One highly related direction is to model and estimate 3D hand joints [5,44,45,53,60,63,66,72,78,79,80,85,3] and mesh [26,27,51,55,75,83,46], object pose [14,28,29,35,43,57,64,71,76,81], or both [6,17,18,19,69].”
Reconstructing Hand-Object Interactions in the Wild (2020, methodology): “We follow the same testing split as [14] for comparison.”
Reconstructing Hands in 3D with Transformers (2023, background): “The improvements in 3D hand pose estimation have also lead to progress in related problems, including joint hand pose and object reconstruction [21, 22, 54, 58] and reconstruction of two interacting hands [28, 32, 37, 38, 46, 55, 60, 61, 65].”
DexYCB: A Benchmark for Capturing Hand Grasping of Objects (2021, methodology): “Regularization Following [24, 13], we add an ℓ2 regularization to the low-dimensional pose embedding of MANO to avoid irregular articulation of hands: Ereg(P ) = 1 NH NH∑ h=1 ||θh|| 2 2 .”
Towards Unconstrained Joint Hand-Object Reconstruction From RGB Videos (2021, methodology): “Hasson 2020 [18], Liu 2021 [41] and our method predict the hand-object meshes jointly.”
ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation (2022, methodology): “[24] Yana Hasson, Bugra Tekin, Federica Bogo, Ivan Laptev, Marc Pollefeys, and Cordelia Schmid.”
Grasping Field: Learning Implicit Representations for Human Grasps (2020, background): “Recently, Hasson et al. [5, 77] achieved promising results on explicitly modeling the contact by combining a parametric hand model, MANO [35], with the mesh based representation for the object.”
H2O: Two Hands Manipulating Objects for First Person Interaction Recognition (2021, methodology): “hand-object pose estimation methods of [33, 80], for comparison purposes, in Table 4 and further provide qualitative examples of our pose predictions in Fig.”