Differentiable vector graphics rasterization for editing and learning

By Tzu-Mao Li, Michal Lukác, Michael Gharbi, Jonathan Ragan-Kelley. 2020

340 citations recorded by the source corpus.

We introduce a differentiable rasterizer that bridges the vector graphics and raster image domains, enabling powerful raster-based loss functions, optimization procedures, and machine learning techniques to edit and generate vector content. We observe that vector graphics rasterization is differentiable after pixel prefiltering. Our differentiable rasterizer offers two prefiltering options: an analytical prefiltering technique and a multisampling anti-aliasing technique. The analytical variant is faster but can suffer from artifacts such as conflation. The multisampling variant is still efficient, and can render high-quality images while computing unbiased gradients for each pixel with respect to curve parameters. We demonstrate that our rasterizer enables new applications, including a vector graphics editor guided by image metrics, a painterly rendering algorithm that fits vector primitives to an image by minimizing a deep perceptual loss function, new vector graphics editing algorithms that exploit well-known image processing methods such as seam carving, and deep generative models that generate vector content from raster-only supervision under a VAE or GAN training objective.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders (2021, methodology): “Drawings in CLIPDraw are represented by a set of differentiable RGBA B ´ ezier curves, following the method by Li et al. (2020).”
Im2Vec: Synthesizing Vector Graphics without Vector Supervision (2021, methodology): “By recombining these primitive paths through a differentiable rasterizer [22] and differentiable compositing [28], we can natively represent vector art while learning to generate it purely based on appearance, obviating the need for vector supervision.”
Modern Evolution Strategies for Creativity: Fitting Concrete Images and Abstract Concepts (2021, methodology): “CLIPDraw [11] proposes to generate the images with diffvg [30], a differentiable SVG renderer.”
ClipGen: A Deep Generative Model for Clipart Vectorization and Synthesis (2021, methodology): “We also adopt a rendering accuracy loss based on a novel differentiable vector graphics rendering technique [6], which encourages the network to reﬁne the predicted geometry and predict the appropriate ﬁlling color simultaneously.”
Generative Art Using Neural Visual Grammars and Dual Encoders (2021, methodology): “First we evolve images, rather than using backpropogation to generate them, as in a recent paper which backpropogated through a diﬀerentiable Scalable Vector Graphics (SVG) system [11].”
StyleCLIPDraw: Coupling Content and Style in Text-to-Drawing Synthesis (2021, positive): “The brush strokes are rendered into a raster image via differentiable model [3].”