Adversarial Inverse Graphics Networks: Learning 2D-to-3D Lifting and Image-to-Image Translation from Unpaired Supervision

By Hsiao-Yu Fish Tung, Adam W. Harley, William Seto, Katerina Fragkiadaki. 2017

163 citations recorded by the source corpus.

Researchers have developed excellent feed-forward models that learn to map images to desired outputs, such as to the images’ latent factors, or to other images, using supervised learning. Learning such mappings from unlabelled data, or improving upon supervised models by exploiting unlabelled data, remains elusive. We argue that there are two important parts to learning without annotations: (i) matching the predictions to the input observations, and (ii) matching the predictions to known priors. We propose Adversarial Inverse Graphics networks (AIGNs): weakly supervised neural network models that combine feedback from rendering their predictions, with distribution matching between their predictions and a collection of ground-truth factors. We apply AIGNs to 3D human pose estimation and 3D structure and egomotion estimation, and outperform models supervised by only paired annotations. We further apply AIGNs to facial image transformation using super-resolution and inpainting renderers, while deliberately adding biases in the ground-truth datasets. Our model seamlessly incorporates such biases, rendering input faces towards young, old, feminine, masculine or Tom Cruiselike equivalents (depending on the chosen bias), or adding lip and nose augmentations while inpainting concealed lips and noses.

Subfields: Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs (2017, background): “Various methods have also been proposed to learn an image-to-image translation in the absence of training pairs [2, 33, 34, 46, 49, 51, 54, 63].”
End-to-End Recovery of Human Shape and Pose (2017, methodology): “Even methods that are based on a reprojection loss [33, 47, 50] require paired 2D-to-3D training data.”
Synthesizing Programs for Images using Reinforced Adversarial Learning (2018, methodology): “, 2017) and AIGN (Tung et al., 2017) fall in this category.”
3D Human Pose Estimation in Video With Temporal Convolutions and Semi-Supervised Training (2018, methodology): “[56] use GANs to learn from unpaired 2D/3D datasets and include a 2D projection consistency term.”
Exposure (2017, background): “The Adversarial Inverse Graphics Network (AIGN) (Tung et al. 2017), by utilizing a problem-specific renderer, can make use of unpaired data for image-to-image translation as well.”
Self-Supervised Learning of 3D Human Pose Using Multi-View Geometry (2019, methodology): “[41] uses unpaired 3D supervision which is easier to get.”
Augmented CycleGAN: Learning Many-to-Many Mappings from Unpaired Data (2018, background): “This framework has been shown to learn convincing mappings across image domains and proved successful in a variety of related applications (Tung et al., 2017; Wolf et al., 2017; Hoffman et al., 2017).”
HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimation (2020, methodology): “Two-stage approaches ﬁrst estimate 2D pose and then lift them to 3D joint locations by a learned dictionary of 3D skeleton [1, 55, 66, 58, 78, 79] or regression [48, 73, 42, 11, 34, 61].”
Learning to Segment via Cut-and-Paste (2018, background): “More generally, and as in work such as [29], we could envisage a range of settings where vision+graphics imitate photography, with adversarial losses to jointly optimise image understanding and rendering stages.”
Image-to-Image Translation: Methods and Applications (2021, methodology): “I2I methods can also contribute to other visual tasks, such as transfer learning for reinforcement learning [217], image registration [38], domain adaptation [29], [30], [31], person re-identification [218], [219], [220], [221], [222], image segmentation [7], [8], [9], facial geometry reconstruction”
C3DPO: Canonical 3D Pose Networks for Non-Rigid Structure From Motion (2019, methodology): “Besides the fully supervised methods [25, 26], several works have explored multi-view supervision [20, 29, 31], ordinal depth supervision [28], unpaired 2D-3D data [30, 36, 41, 15] or videos [17] to alleviate the need for full 2D-3D annotations.”
Visual Object Networks: Image Generation with Disentangled 3D Representations (2018, background): “…inverse graphics” [Yuille and Kersten, 2006, Bever and Poeppel, 2010], researchers have made much progress in recent years on learning to invert graphics engines, many with deep neural networks [Kulkarni et al., 2015b, Yang et al., 2015, Kulkarni et al., 2015a, Tung et al., 2017, Shu et al., 2017].”
Deep 3D human pose estimation: A review (2021, unknown): “To produce a more realistic 3D human pose, generative adversarial networks (GANs) are often used (Fish Tung et al., 2017; Yang et al., 2018; Wandt and Rosenhahn, 2019).”
Self-supervised Learning of Motion Capture (2017, background): “Deep geometry learning Our differentiable renderer follows recent works that integrate deep learning and geometric inference [33].”
Unsupervised Learning of Probably Symmetric Deformable 3D Objects From Images in the Wild (2019, result): “We also compare to the baselines MOFA [53] and AIGN [57] reported in [37].”