Learning to Sketch with Shortcut Cycle Consistency

By Jifei Song, Kaiyue Pang, Yi-Zhe Song, Tao A. Xiang, Timothy M. Hospedales. 2018

137 citations recorded by the source corpus.

To see is to sketch - free-hand sketching naturally builds ties between human and machine vision. In this paper, we present a novel approach for translating an object photo to a sketch, mimicking the human sketching process. This is an extremely challenging task because the photo and sketch domains differ significantly. Furthermore, human sketches exhibit various levels of sophistication and abstraction even when depicting the same object instance in a reference photo. This means that even if photo-sketch pairs are available, they only provide weak supervision signal to learn a translation model. Compared with existing supervised approaches that solve the problem of D(E(photo)) â†’ sketch), where E(Â·) and D(Â·) denote encoder and decoder respectively, we take advantage of the inverse problem (e.g., D(E(sketch) â†’ photo), and combine with the unsupervised learning tasks of within-domain reconstruction, all within a multi-task learning framework. Compared with existing unsupervised approaches based on cycle consistency (i.e., D(E(D(E(photo)))) â†’ photo), we introduce a shortcut consistency enforced at the encoder bottleneck (e.g., D(E(photo)) â†’ photo) to exploit the additional self-supervision. Both qualitative and quantitative results show that the proposed model is superior to a number of state-of-the-art alternatives. We also show that the synthetic sketches can be used to train a better fine-grained sketch-based image retrieval (FG-SBIR) model, effectively alleviating the problem of sketch data scarcity.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketches (2022, background): “Moreover, due to its highly abstract nature, the same object may be sketched in various ways under unique user-styles [51, 46], and with varied levels of detail [45].”
Partially Does It: Towards Scene-Level FG-SBIR with Partial Input (2022, background): “Related Work Fine-grained sketch based image retrieval The ability of sketches to offer inherently fine-grained visual descriptions commences the avenues of fine-grained sketch-based image retrieval (FG-SBIR) [4, 7, 8, 52, 53, 57, 62].”
Cross-Modal Hierarchical Modelling for Fine-Grained Sketch Based Image Retrieval (2020, methodology): “Datasets: We evaluate our method on FG-SBIR specific datasets of QMUL-Chair-V2 [40] and QMUL-Shoe-V2 [30, 40], along with SWIRE [15] dataset which was curated using GUI examples from the RICO [8] dataset.”
Sketchformer: Transformer-Based Representation for Sketched Structure (2020, background): “Variants of CycleGAN [32] have also shown to be useful as generative models for sketch [33].”
StyleMeUp: Towards Style-Agnostic Sketch-Based Image Retrieval (2021, methodology): “It adapts using inner loop updates across retrieval tasks over categories in SBIR and over instances in FG-SBIR frameworks”
Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting (2021, methodology): “(iii) We use deterministic cross-modal encoder-decoder architecture since VAE-based design [57] reduces the performance.”
Sketch Less for More: On-the-Fly Fine-Grained Sketch-Based Image Retrieval (2020, unknown): “Datasets: We use QMUL-Shoe-V2 [29, 34, 40] and QMUL-Chair-V2 [40] datasets that have been specifically designed for FG-SBIR.”
Sketch3T: Test-Time Training for Zero-Shot SBIR (2022, background): “Research thus advanced to finegrained SBIR [7, 38, 52] beginning with deformable-part models [25].”
More Photos are All You Need: Semi-Supervised Learning for Fine-Grained Sketch Based Image Retrieval (2021, methodology): “Datasets: Two publicly available datasets, QMUL-Shoe-V2 [28, 33, 41, 5] and QMUL-Chair-V2 [5, 41] are Get reward R G using F and D C .”
Deep Learning for Free-Hand Sketch: A Survey (2020, background): “ing both: (i) sketch-to-photo generation [17], [273], [274], (ii) photo-to-sketch generation [55], [271], [272], [275].”
SketchEmbedNet: Learning Novel Concepts by Imitating Drawings (2020, methodology): “This is a novel setting, as prior efforts have only applied their sketch RNN models on the Quickdraw dataset or natural images with only two object classes (shoe/chair) and scrubbed backgrounds (Chen et al., 2017; Ha and Eck, 2018; Song et al., 2018; Yu et al., 2016).”
Creative Sketch Generation (2020, background): “Earlier studies generated a sketch from an image via an image-to-image translation approach (Isola et al., 2017; Song et al., 2018; Li et al., 2019b).”