Synthesizing Programs for Images using Reinforced Adversarial Learning

By Yaroslav Ganin, Tejas D. Kulkarni, Igor Babuschkin, Ali Eslami, Oriol Vinyals. 2018

243 citations recorded by the source corpus.

Advances in deep generative networks have led to impressive results in recent years. Nevertheless, such models can often waste their capacity on the minutiae of datasets, presumably due to weak inductive biases in their decoders. This is where graphics engines may come in handy since they abstract away low-level details and represent images as high-level programs. Current methods that combine deep learning and renderers are limited by hand-crafted likelihood or distance functions, a need for large amounts of supervision, or difficulties in scaling their inference algorithms to richer datasets. To mitigate these issues, we present SPIRAL, an adversarially trained agent that generates a program which is executed by a graphics engine to interpret and sample images. The goal of this agent is to fool a discriminator network that distinguishes between real and rendered data, trained with a distributed reinforcement learning setup without any supervision. A surprising finding is that using the discriminator's output as a reward signal is the key to allow the agent to make meaningful progress at matching the desired output rendering. To the best of our knowledge, this is the first demonstration of an end-to-end, unsupervised and adversarial inverse graphics agent on challenging real world (MNIST, Omniglot, CelebA) and synthetic 3D datasets.

Subfields: Artificial Intelligence, Biophysics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biophysics

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning to Understand Goal Specifications by Modelling Reward (2018, background): “Several recent imitation learning works consider using goal-states directly for deﬁning the task (Ganin et al., 2018; Pathak et al., 2018).”
Structured agents for physical construction (2019, background): “In typical RL and control settings that involve placing objects, the agent takes absolute actions in the frame of reference of the observation (e.g. Silver et al., 2016; 2018; Zhang et al., 2018; Ganin et al., 2018; Janner et al., 2019).”
Causal Reasoning from Meta-reinforcement Learning (2019, background): “…more complex episodes, as well as models which are more explicitly compositional (e.g. Andreas et al., 2016; Battaglia et al., 2018) or have richer semantics (e.g. Ganin et al., 2018) that can more explicitly leverage symmetries in the environment and improve generalization and training efficiency.”
Automatically Composing Representation Transformations as a Means for Generalization (2018, background): “The ﬁrst as-Automatically sumes pre-speciﬁed transformations and learns the structure (from dense supervision on execution traces to sparse-rewards) [13, 14, 18, 22, 25, 28, 65, 87, 90].”
Differentiable vector graphics rasterization for editing and learning (2020, methodology): “Ganin et al. [2018] instead trained a model-free reinforcement learning agent on natural images using a drawing simulator, combined with an adversarial loss.”
Exploiting Hierarchy for Learning and Transfer in KL-regularized RL (2019, background): “Recent advances have greatly improved data efﬁciency, scalability, and stability of these algorithms in a variety of domains (Rennie et al., 2017; Zoph & Le, 2017; Espeholt et al., 2018; Ganin et al., 2018; Zhu et al., 2018).”
Learning to Follow Language Instructions with Adversarial Reward Induction (2018, background): “Several recent imitation learning works considered using goal-states directly for defining the task [10, 21].”
Beyond imitation: Zero-shot task transfer on robots by learning concepts as cognitive programs (2018, background): “ing. Such transfer requires visual analogy making (41,42) as opposed to rote imitation. While we take inspiration from earlier works on learning programs to describe the contents of images and videos (38,43,44), those works focused on settings where the content to be described was fully observable i”
Learning to Infer and Execute 3D Shape Programs (2019, background): “The very recent SPIRAL system (Ganin et al., 2018) also takes as its goal to learn structured program–like models from (2D) images.”
Im2Vec: Synthesizing Vector Graphics without Vector Supervision (2021, background): “Ganin et al. [10], Huang et al. [18], and Nakano [25] train Reinforcement Learning (RL) drawing agents.”
Meta-Sim2: Unsupervised Learning of Scene Structure for Synthetic Data Generation (2020, background): “[19,47] train an agent to paint images using brush strokes in an adversarial setting with Reinforcement Learning.”
The Omniglot Challenge: A 3-Year Progress Report (2019, background): “Many new approaches produce high-quality unconditional samples (Gregor et al., 2015; Eslami et al., 2016; Gregor et al., 2016; Ganin et al., 2018), although they have not been evaluated for their generative creativity, as opposed to merely copying characters in the training set.”
DreamCoder: bootstrapping inductive program synthesis with wake-sleep library learning (2021, background): “Existing neural program synthesis systems pair a fixed DSL to a learned neural network that guides program search [4, 15, 18, 38, 42].”
Deep Learning for Free-Hand Sketch: A Survey (2020, background): “Another line of work within sketch generation uses differentiable rendering [56], [130], [138] or reinforcement learning [122], [125], [131], [139], [140] to train policies that”
Execution-Guided Neural Program Synthesis (2018, background): “Synthesizing a program from input-output examples is an important challenging problem with many applications (Devlin et al., 2017b; Gulwani et al., 2012; Gulwani, 2011; Bunel et al., 2018; Chen et al., 2018; Cai et al., 2017; Li et al., 2017; Reed & De Freitas, 2016; Zaremba et al., 2016; Zaremba & ”
Learning to Infer Graphics Programs from Hand-Drawn Images (2017, background): “The very recent SPIRAL [18] system learns to infer procedures for controlling a ‘pen’ to derender highly diverse natural images, complementing our focus here on more abstract procedures but less natural images.”
A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications (2020, methodology): “Furthermore, GANs were combined with reinforcement learning for synthesizing programs for images [266].”