Weakly-supervised Disentangling with Recurrent Transformations for 3D View Synthesis

By Jimei Yang, Scott E. Reed, Ming-Hsuan Yang, Honglak Lee. 2015

317 citations recorded by the source corpus.

An important problem for both graphics and vision is to synthesize novel views of a 3D object from a single image. This is particularly challenging due to the partial observability inherent in projecting a 3D object onto the image space, and the ill-posedness of inferring object shape and pose. However, we can train a neural network to address the problem if we restrict our attention to specific object categories (in our case faces and chairs) for which we can gather ample training data. In this paper, we propose a novel recurrent convolutional encoder-decoder network that is trained end-to-end on the task of rendering rotated objects starting from a single image. The recurrent structure allows our model to capture long-term dependencies along a sequence of transformations. We demonstrate the quality of its predictions for human faces on the Multi-PIE dataset and for a dataset of 3D chair models, and also show its ability to disentangle latent factors of variation (e.g., identity and pose) without using full supervision.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets (2016, background): “[20] developed a recurrent variant that generates a sequence of latent factor transformations.”
A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation (2020, background): “…perform well on realistic settings most of the other works have considered some more or less explicit form of supervision (Reed et al., 2014; Zhu et al., 2014; Yang et al., 2015; Kulkarni et al., 2015; Cheung et al., 2014; Mathieu et al., 2016; Narayanaswamy et al., 2017; Suter et al., 2019).”
Generative Adversarial Text to Image Synthesis (2016, background): “Yang et al. (2015) added an encoder network as well as actions to this approach.”
StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks (2016, background): “Conditioned on given text descriptions, conditional- GANs [26, 24] are able to generate images that are highly related to the text meanings.”
Action-Conditional Video Prediction using Deep Networks in Atari Games (2015, neutral): “This result implies that our model learns to distinguish between controlled objects and uncontrolled objects and transform them using disentangled representations (see [25, 24, 37] for related work on disentangling factors of variation).”
Stacked Hourglass Networks for Human Pose Estimation (2016, methodology): “[46] employ an encoder-decoder architecture without skip connections for image generation.”
Dynamic Filter Networks (2016, background): “[23] learn to rotate a given face to another pose.”
Weakly-Supervised Disentanglement Without Compromises (2020, background): “Initially, the community focused on more or less explicit and task dependent supervision (Reed et al., 2014; Yang et al., 2015; Kulkarni et al., 2015; Cheung et al., 2014; Mathieu et al., 2016; Narayanaswamy et al., 2017).”
On the Fairness of Disentangled Representations (2019, background): “Disentangling pose from content and content from motion are also classical computer vision problems that have been tackled with various degrees of supervision and inductive bias [92, 93, 40, 25, 18, 31, 42].”
Are Disentangled Representations Helpful for Abstract Visual Reasoning? (2019, background): “Disentangling pose, style, or motion from content are classical vision tasks that has been studied with different degrees of supervision [70, 76, 77, 34, 19, 14, 21, 36].”
Towards a Definition of Disentangled Representations (2018, background): “However, many other approaches [50, 23, 66, 63, 80, 78, 36, 17, 77, 46] and a recent disentanglement metric by [65] consider it perfectly ﬁne for each data generative factor to be encoded by multiple latent dimensions.”
Variational Inference of Disentangled Latent Concepts from Unlabeled Observations (2017, methodology): “…the earlier work assumes some sort of supervision in terms of: (i) partial or full access to the generative factors per instance (Reed et al., 2014; Yang et al., 2015; Kulkarni et al., 2015; Karaletsos et al., 2015), (ii) knowledge about the nature of generative factors (e.g, translation,…”
Understanding disentangling in β-VAE (2018, background): “Most initial attempts to learn disentangled representations required supervised knowledge of the data generative factors [18, 35, 32, 44, 43, 12, 24, 7, 42, 20].”
DARLA: Improving Zero-Shot Transfer in Reinforcement Learning (2017, background): “Learning how to project raw observations into such a factorised description of the world is addressed by the large body of literature on disentangled representation learning (Schmidhuber, 1992; Desjardins et al., 2012; Cohen & Welling, 2014; 2015; Kulkarni et al., 2015; Hinton et al., 2011; Rippel &”
Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision (2016, background): “Learning from transformed sensory data has gained attention [12, 5, 15, 13, 23, 6, 24] in recent years.”
The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision (2019, background): “Kulkarni et al. (2015) proposed convolutional inverse graphics networks for learning and inferring pose of faces, while Yang et al. (2015) learned disentangled representation of pose of chairs from images.”
Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding (2018, neutral): “Our work is closely related to research on learning an inter-pretable, disentangled representation with a neural network [Kulkarni et al., 2015, Yang et al., 2015, Wu et al., 2017].”
Multi-view 3D Models from Single Images with a Convolutional Network (2015, background): “Most closely related is the concurrent work by Yang et al. [13,14].”