Learning to Infer Graphics Programs from Hand-Drawn Images

By Kevin Ellis, Daniel Ritchie, Armando Solar-Lezama, Josh Tenenbaum. 2017

252 citations recorded by the source corpus.

We introduce a model that learns to convert simple hand drawings into graphics programs written in a subset of \LaTeX. The model combines techniques from deep learning and program synthesis. We learn a convolutional neural network that proposes plausible drawing primitives that explain an image. These drawing primitives are like a trace of the set of primitive commands issued by a graphics program. We learn a model that uses program synthesis techniques to recover a graphics program from that trace. These programs have constructs like variable bindings, iterative loops, or simple kinds of conditionals. With a graphics program in hand, we can correct errors made by the deep network, measure similarity between drawings by use of similar high-level geometric structures, and extrapolate drawings. Taken together these results are a step towards agents that induce useful, human-readable programs from perceptual input.

Subfields: Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

Citations in the corpus

A Survey of Machine Learning for Big Code and Naturalness (2017, neutral): “[55] designed multimodal model that accept visual input (the non-code modality) and generate code in a DSL describing how the input (hand-drawn image, GUI screenshot) was constructed.”
DeepProbLog: Neural Probabilistic Logic Programming (2018, methodology): “[49] generate a LATEX code from a hand-drawn sketch by using a neural network to recognise basic shapes within”
Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions (2018, background): “, 2016; Reed & De Freitas, 2015), visual understanding (Ellis et al., 2017; Johnson et al., 2017), and natural language processing (Andreas et al.”
Synthesizing Programs for Images using Reinforced Adversarial Learning (2018, positive): “Ellis et al. (2017) proposed a visual program induction model to infer L A TEXprograms for diagram understanding.”
DreamCoder: growing generalizable, interpretable knowledge with wake–sleep Bayesian program learning (2020, background): “eal-world data is far messier. A key challenge for program induction going forward is to handle more pervasive noise and uncertainty, by leaning more heavily on probabilistic and neural AI approaches (5,43,44). Recent research has explored program induction with various hybrid neuro-symbolic represe”
Deep Learning for Free-Hand Sketch: A Survey (2020, methodology): “By considering an interpretable latent representation of sketches, such methods can also potentially be used to de-render sketches into programs or symbols [141], [142].”
HOUDINI: Lifelong Learning as Program Synthesis (2018, background): “A few recent approaches to program synthesis do combine symbolic and neural methods [11, 6, 12, 28, 18].”
Differentiable vector graphics rasterization for editing and learning (2020, background): “Ellis et al. [2018] combined program synthesis and recurrent neural networks to infer a graphics program that generates a target sketch image.”
Leveraging Language to Learn Program Abstractions and Search Heuristics (2021, background): “…et al., 2017), semantic parsing (Artzi & Zettlemoyer, 2013; Liang, 2016), structured visual understanding (Johnson et al., 2017b; Yi et al., 2018), image generation (Ellis et al., 2017; Ganin et al., 2018), and policy learning (Fikes & Nilsson, 1971; Cropper & Muggleton, 2015; Silver et al., 2020).”
Im2Vec: Synthesizing Vector Graphics without Vector Supervision (2021, methodology): “Ellis et al. [8] use program synthesis to generate graphics expressed using a subset of the L A TEX language from hand drawings.”
Improving Neural Program Synthesis with Inferred Execution Traces (2018, methodology): “In Ellis et al. [2017], the authors first infer the execution trace of a drawing program and leverages a generic program search algorithm on the trace.”
Playgol: learning programs through play (2019, methodology): “In other words, our approach allows a learner to discover highly reusable concepts without a user-supplied corpus of training tasks, which Ellis et al. [Ellis et al. , 2018a] argue is essential for program induction to become a standard part of the AI toolkit.”
Write, Execute, Assess: Program Synthesis with a REPL (2019, background): “Integrating learning and search to tackle the problem of program synthesis is an old idea experiencing a recent resurgence [2, 3, 4, 5, 6, 7, 8, 9, 10, 11].”
Neurosymbolic Programming (2021, methodology): “A second category of methods, which we call learning to specify (Ellis et al., 2018b), determine how to generalize incomplete or ambiguous task specifications to more complete specifications.”
DeepSVG: A Hierarchical Generative Network for Vector Graphics Animation (2020, methodology): “Unlike raster graphics, vector graphics generation has not received extensive attention yet, and has been mostly limited to high-level shape synthesis [2] or sketch generation, using the ‘Quick, Draw!’ [19] dataset.”
Learning Compositional Neural Programs with Recursive Tree Search and Planning (2019, background): “Ellis et al. [2018] shows how to generate graphics programs to reproduce hand drawn images.”