Sketchformer: Transformer-Based Representation for Sketched Structure

By Leo Sampaio Ferraz Ribeiro, Tu Bui, John Philip Collomosse, Moacir Antonelli Ponti. 2020

154 citations recorded by the source corpus.

Sketchformer is a novel transformer-based representation for encoding free-hand sketches input in a vector form, i.e. as a sequence of strokes. Sketchformer effectively addresses multiple tasks: sketch classification, sketch based image retrieval (SBIR), and the reconstruction and interpolation of sketches. We report several variants exploring continuous and tokenized input representations, and contrast their performance. Our learned embedding, driven by a dictionary learning tokenization scheme, yields state of the art performance in classification and image retrieval tasks, when compared against baseline representations driven by LSTM sequence to sequence architectures: SketchRNN and derivatives. We show that sketch reconstruction and interpolation are improved significantly by the Sketchformer embedding for complex sketches with longer stroke sequences.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketches (2022, methodology): “For learning the joint embedding space, category-level SBIR typically employs either CNN [11, 12], RNN [65], or Transformer [41] based Siamese networks, accompanied by a triplet-ranking objective [68].”
Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting (2021, unknown): “For Transformer like encoders, we take input via a trainable linear layer to convert each five element vector to the Transformer’s model dimension.”
Sketching without Worrying: Noise-Tolerant Sketch-Based Image Retrieval (2022, methodology): “Like any other retrieval system, Deep Neural Networks have become a de-facto choice for any recent SBIR frameworks [15, 13, 37, 60, 12, 7] over early hand-engineered feature descriptors [50].”
DeepSVG: A Hierarchical Generative Network for Vector Graphics Animation (2020, methodology): “(b) DeepSVG (ours) Figure 2: One-stage autoregressive autoencoder architectures [5, 11, 17] (a) take the entire draw commands as input and decode the latent vector one command at a time.”
Cross-Modal Hierarchical Modelling for Fine-Grained Sketch Based Image Retrieval (2020, methodology): “the sequential [13], abstract [26], and stroke-wise [1] characteristics of sketches, either separately or in combination [33, 48].”
SketchEmbedNet: Learning Novel Concepts by Imitating Drawings (2020, background): “…generative adversarial networks (GANs) (Isola et al., 2017; Li et al., 2019), to autoregressive (Gregor et al., 2015; Ha & Eck, 2018; Chen et al., 2017), transformer (Ribeiro et al., 2020; Aksan et al., 2020), hierarchical Bayesian (Lake et al., 2015) and neuro-symbolic (Tian et al., 2020) models.”
More Photos are All You Need: Semi-Supervised Learning for Fine-Grained Sketch Based Image Retrieval (2021, background): “With the ever rising popularity of touch screen devices, sketch-based image retrieval (SBIR) has witnessed significant interest within the vision community [5, 34, 35, 50, 11, 45].”
Deep Learning for Free-Hand Sketch: A Survey (2020, background): “The advantage of sketch queries will become more signiﬁcantly when users want to search something in ﬁne-grained/instance-level.”
Creative Sketch Generation (2020, background): “, 2018), sketch-based image retrieval (Yu et al., 2016; Liu et al., 2017; Ribeiro et al., 2020) and generation (Gao et al.”
Im2Vec: Synthesizing Vector Graphics without Vector Supervision (2021, methodology): “Similarly, Sketchformer [29] proposed a transformer based architecture for encoding vector form sketches.”