A Neural Representation of Sketch Drawings

By David R Ha, Douglas Eck. 2017

996 citations recorded by the source corpus.

We present sketch-rnn, a recurrent neural network (RNN) able to construct stroke-based drawings of common objects. The model is trained on thousands of crude human-drawn images representing hundreds of classes. We outline a framework for conditional and unconditional sketch generation, and describe new robust training methods for generating coherent sketch drawings in a vector format.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Control and Systems Engineering, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Representation & Self-Supervised Learning

Citations in the corpus

Recurrent World Models Facilitate Policy Evolution (2018, methodology): “During sampling, we can adjust a temperature parameter τ to control model uncertainty, as done in previous work [28].”
Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples (2019, background): “While some of the datasets that were added to the meta-training phase did see an improvement across all models, in particular for Omniglot and Quick Draw, this was not true across the board.”
Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis (2018, methodology): “The two conditionals are modeled by a Gaussian Mixture Model (GMM) and a categorical distribution [8], respectively:”
Global-and-local attention networks for visual recognition (2018, background): “We expect that a more systematic analysis – beyond identifying what features are selected but also when they are selected [45, 46] could help further our understanding of the different attention mechanisms responsible for the selection of diagnostic image features.”
Probing Representation Forgetting in Supervised and Unsupervised Continual Learning (2022, background): “Finally though our work studies a diverse task sequence ImageNet → Scenes → CUB → Flowers, to fully understand the behavior of representation forgetting, results over more distant tasks may be needed (e.g. ImageNet → Sketch Images [13,44])”
Text-to-Image Generation Grounded by Fine-Grained User Attention (2020, background): “Such interfaces would support various applications, allowing users to rapidly create interactive scenes without specialized software [6], assist the creative process of an artist [13], or generate realistic images that are more closely grounded to intended descriptions [33].”
Synthesizing Programs for Images using Reinforced Adversarial Learning (2018, methodology): “More recently, the sketch-rnn model (Ha & Eck, 2017) used sequence-to-sequence learning (Sutskever et al., 2014) to produce impressive sketches both unconditionally and conditioned on data.”
PointCNN: Convolution On X-Transformed Points (2018, methodology): “PointCNN outperforms PointNet++ on both datasets, with a more prominent advantage on Quick Draw (25M data samples), which is signiﬁcantly larger than TU-Berlin (0.02M data samples).”
Sketch-Guided Object Localization in Natural Images (2020, positive): “, SketchRNN [9], transformer-based architecture [36].”
VectorMapNet: End-to-end Vectorized HD Map Learning (2022, methodology): “These methods typically treat geometric elements as a sequence, such as primitive parts of furniture (Li et al., 2017; Mo et al., 2019), states of sketch strokes (Ha & Eck, 2017), vertices of n -gon mesh (Nash et al., 2020), and parameters of SVG primitives (Carlier et al., 2020).”
Sketch3T: Test-Time Training for Zero-Shot SBIR (2022, background): “Research efforts have mainly focused on addressing the sketch-photo domain gap, incurred by abstraction [33], drawing style [47] and stroke saliency [19].”
Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting (2021, methodology): “For RNN-like architectures [22], we feed the five elements vector vt at every time step, and take the hidden state of final time step as the encoded latent representation.”
Machine learning in protein engineering (2018, background): “Tantalizingly, generative models in other ﬁelds have been trained to generate new faces (Radford et al., 2015), sketches (Ha & Eck, 2017), and even music (Roberts et al., 2018).”
SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis (2018, methodology): “The newly published QuickDraw dataset [16] has an impressive 50 million sketches.”
Biases in Generative Art: A Causal Look from the Lens of Art History (2020, background): “In [Ha and Eck, 2018], a recurrent neural network is proposed to construct stroke-based drawings of common objects.”
Creative Sketch Generation (2020, methodology): “One of the early works in this direction was SketchRNN (Ha & Eck, 2018) – a sequence-to-sequence Variational Autoencoder (VAE) that models the temporal dependence among strokes in a sketch.”