Disentangling Latent Hands for Image Synthesis and Pose Estimation

By Linlin Yang, Angela Yao. 2018

124 citations recorded by the source corpus.

Hand image synthesis and pose estimation from RGB images are both highly challenging tasks due to the large discrepancy between factors of variation ranging from image background content to camera viewpoint. To better analyze these factors of variation, we propose the use of disentangled representations and a disentangled variational autoencoder (dVAE) that allows for specific sampling and inference of these factors. The derived objective from the variational lower bound as well as the proposed training strategy are highly flexible, allowing us to handle crossmodal encoders and decoders as well as semi-supervised learning scenarios. Experiments show that our dVAE can synthesize highly realistic images of the hand specifiable by both pose and image background content and also estimate 3D hand poses from RGB images with accuracy competitive with state-of-the-art on two public benchmarks.

Subfields: Computer Vision and Pattern Recognition, Generative Models, 3D Vision & Pose, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In 3D Vision & Pose

In Human-Computer Interaction

Citations in the corpus

Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications (2022, background): “One highly related direction is to model and estimate 3D hand joints [5,44,45,53,60,63,66,72,78,79,80,85,3] and mesh [26,27,51,55,75,83,46], object pose [14,28,29,35,43,57,64,71,76,81], or both [6,17,18,19,69].”
Two-hand Global 3D Pose Estimation using Monocular RGB (2020, background): “[33] proposed to use variational autoencoders to learn a latent space for hand poses, which are capable of estimating 3D hand poses from RGB inputs.”
Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wild (2020, background): “In contrast, variational approaches learn a shared latent space by an alternated training between variational autoencoders [46], disentanglement of the latent representation [57], or the alignment of latent spaces [56].”
InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Image (2020, background): “[41] proposed deep generative models to learn latent space for hand.”
LaDDer: Latent Data Distribution Modelling with a Generative Prior (2020, unknown): “Thanks to all the preferable properties, VAEs have been widely applied in many computer vision applications, including image synthesis [11, 17, 41], human motion modelling [14, 36] and 3D reconstruction [1, 4, 6, 10, 29].”
Knowledge As Priors: Cross-Modal Knowledge Generalization for Datasets Without Superior Knowledge (2020, methodology): “We compare the 3D PCK curves with state-of-the-art methods [5, 40, 52, 53, 59] on RHD dataset in Fig.”
StyleMeUp: Towards Style-Agnostic Sketch-Based Image Retrieval (2021, unknown): “Starting from generic frameworks like combining autoencoders with adversarial training [32], this disentanglement paradigm has been successfully applied to recognition [55, 38], image-to-image translation [58] and imageediting [22, 56] tasks.”
Reconstructing Hand-Object Interactions in the Wild (2020, background): “Many previous works on hand pose estimation directly predict 3D joint locations from either depth [50, 53, 55, 59, 64, 65, 35] or RGB [45, 36, 4, 63, 68] images.”
Leveraging Photometric Consistency Over Time for Sparsely Supervised Hand-Object Reconstruction (2020, unknown): “Several approaches for hand pose estimation from RGB images focus on regressing 3D skeleton joint positions [5, 17, 26, 44, 52, 55].”
FrankMocap: Fast Monocular 3D Hand and Body Motion Capture by Regression and Integration (2020, unknown): “Recent work begins to use single RGB images as input [62, 32, 9, 57].”