Neural Scene De-rendering

By Jiajun Wu, Josh Tenenbaum, Pushmeet Kohli. 2017

160 citations recorded by the source corpus.

We study the problem of holistic scene understanding. We would like to obtain a compact, expressive, and interpretable representation of scenes that encodes information such as the number of objects and their categories, poses, positions, etc. Such a representation would allow us to reason about and even reconstruct or manipulate elements of the scene. Previous works have used encoder-decoder based neural architectures to learn image representations, however, representations obtained in this way are typically uninterpretable, or only explain a single object in the scene. In this work, we propose a new approach to learn an interpretable distributed representation of scenes. Our approach employs a deterministic rendering function as the decoder, mapping a naturally structured and disentangled scene description, which we named scene XML, to an image. By doing so, the encoder is forced to perform the inverse of the rendering operation (a.k.a. de-rendering) to transform an input image to the structured scene XML that the decoder used to produce the image. We use a object proposal based encoder that is trained by minimizing both the supervised prediction and the unsupervised reconstruction errors. Experiments demonstrate that our approach works well on scene de-rendering with two different graphics engines, and our learned representation can be easily adapted for a wide range of applications like image editing, inpainting, visual analogy-making, and image captioning.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Neural Logic Machines (2019, background): “Santoro et al. (2017) implements a visual reasoning system based on “virtual” objects brought by receptive ﬁelds in CNNs. Wu et al. (2017) tackles the problem of deriving structured representation from raw pixel-level inputs.”
Reconciling deep learning with symbolic artificial intelligence: representing objects and relations (2019, background): “In [24], the authors show how a graphics engine can be combined with a deep neural network to force it to learn such representations.”
Towards causal generative scene models via competition of experts (2020, methodology): “Furthermore, gains achieved by a more accurate scene generation process are generally paid for by complicated inference, and most methods thus rely on variations of MCMC sampling schemes (Jampani et al., 2015; Wu et al., 2017).”
CLEVRER: CoLlision Events for Video REpresentation and Reasoning (2019, methodology): “We draw inspirations from Yi et al. (2018) and study an oracle model that operates on a symbolic representation to join video perception, language understanding with dynamics modeling.”
MONet: Unsupervised Scene Decomposition and Representation (2019, background): “Methods in this camp [Tian et al., 2019, Yao et al., 2018] make domain-speciﬁc assumptions about the latent code [Kulkarni et al., 2015] or generative process [Wu et al., 2017] to keep inference tractable, but these assumptions tend to be too strong for the methods to be broadly useful.”
The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision (2019, methodology): “Following Yi et al. (2018), we generate 10,000 Minecraft scenes using the ofﬁcially open-sourced tools by Wu et al. (2017).”
Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding (2018, methodology): “Our work is closely related to research on learning an inter-pretable, disentangled representation with a neural network [Kulkarni et al., 2015, Yang et al., 2015, Wu et al., 2017].”
Image Generation from Scene Graphs (2018, methodology): “Related to our layout prediction, Chang et al . have investigated text to 3D scene generation [4, 5]; other approaches to image synthesis include stochastic grammars [20], prob-abalistic programming [27], inverse graphics [28], neural de-rendering [55], and generative ConvNets [56].”
Synthesizing Programs for Images using Reinforced Adversarial Learning (2018, background): “Structured object-attribute based ‘de-rendering’ models have been proposed for interpretation of images (Wu et al., 2017b) and videos (Wu et al., 2017a).”
Learning to Manipulate Individual Objects in an Image (2020, methodology): “Similarly, [35] employs a graphics engine as the decoder to enforce an interpretable representation, which is, however, not backward-differentiable.”
3D-Aware Scene Manipulation via Inverse Graphics (2018, unknown): “Our work most resembles the paper by Wu et al. [2017a], who propose to ‘de-render’ an image with an encoder-decoder framework that uses a neural network as the encoder and a graphics engine as the decoder.”
Adversarial Geometry and Lighting using a Differentiable Renderer (2018, background): “2017; Zeng et al., 2017), and in generalizing neural style transfer to a 3D context (Kato et al., 2018; Liu et al., 2018). Recently, various techniques have been proposed to obtain these derivatives: Wu et al. (2017); Liu et al. (2017); Eslami et al. (2016) use neural networks to learn the image for”
VectorDefense: Vectorization as a Defense to Adversarial Examples (2018, background): “Instead it might be interesting to de-render an image into a scene graph [41], and train a prior over the graphs [18].”
Generative Modeling of Infinite Occluded Objects for Compositional Scene Representation (2019, unknown): “Layered representations have been studied for years in the computer vision community, and several methods which handle occlusions have been proposed (Wang & Adelson, 1994; Williams & Titsias, 2004; Le Roux et al., 2011; Huang & Murphy, 2016; Moreno et al., 2016; Wu et al., 2017).”
Learning to Infer Graphics Programs from Hand-Drawn Images (2017, methodology): “w contributions of this work are: (1) A model that converts sketches to high-level programs: in contrast to converting images to vectors or low-level parses (Huang et al., 2017; Nishida et al., 2016; Wu et al., 2017; Beltramelli, 2017; Deng et al., 2017). (2) An algorithm for learning a policy for e”
SpreadsheetCoder: Formula Prediction from Semi-structured Context (2021, unknown): “…2017; Bunel et al., 2018; Bavishi et al., 2019; Shin et al., 2018; Chen et al., 2019), natural language descriptions (Gulwani & Marron, 2014; Yu et al., 2018; Yin et al., 2018; Lin et al., 2018; Liang et al., 2018; Wang et al., 2020), and images (Wu et al., 2017; Liu & Wu, 2019; Sun et al., 2018).”