Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting

By Ayan Kumar Bhunia, Pinaki Nath Chowdhury, Yongxin Yang, Timothy M. Hospedales, Tao A. Xiang, Yi-Zhe Song. 2021

69 citations recorded by the source corpus.

Self-supervised learning has gained prominence due to its efficacy at learning powerful representations from unlabelled data that achieve excellent performance on many challenging downstream tasks. However, supervision-free pre-text tasks are challenging to design and usually modality specific. Although there is a rich literature of self-supervised methods for either spatial (such as images) or temporal data (sound or text) modalities, a common pretext task that benefits both modalities is largely missing. In this paper, we are interested in defining a self-supervised pre-text task for sketches and handwriting data. This data is uniquely characterised by its existence in dual modalities of rasterized images and vector coordinate sequences. We address and exploit this dual representation by proposing two novel cross-modal translation pre-text tasks for self-supervised feature learning: Vectorization and Rasterization. Vectorization learns to map image space to vector coordinates and rasterization maps vector coordinates to image space. We show that our learned encoder modules benefit both raster-based and vector-based downstream approaches to analysing hand-drawn data. Empirical evidence shows that our novel pre-text tasks surpass existing single and multi-modal self-supervision methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketches (2022, background): “…41, 46, 34, 53, 6, 2], sketch has also been employed in a variety of vision understanding tasks, including segmentation [22], video synthesis [28], representation learning [60, 3], object localisation [55], image-inpainting [62], 3D shape retrieval [31], 3D shape modelling [71], among others [64].”
Sketch3T: Test-Time Training for Zero-Shot SBIR (2022, methodology): “Here we make clever use of the vectorised nature of sketches, and utilise a self-supervised task of sketchraster to sketch-vector translation [5] to update the featureextractor at inference.”
More Photos are All You Need: Semi-Supervised Learning for Fine-Grained Sketch Based Image Retrieval (2021, background): “Despite starting as a category-level retrieval problem [9, 10, 3, 12], the fine-grained nature of sketches stirred current research focus more towards fine-grained SBIR (FGSBIR) [5, 28] – which aims to retrieve a particular photo based on a query sketch at an intra-category basis.”
StyleMeUp: Towards Style-Agnostic Sketch-Based Image Retrieval (2021, background): “Sketch as an input modality has been proven to be a worthy complement to text for photo image retrieval [8, 10, 4].”
Sketching without Worrying: Noise-Tolerant Sketch-Based Image Retrieval (2022, methodology): “Therefore, taking advantage of the dual representation [5] of the sketch, we model the stroke subset selector on the sequential vector space.”
Deep Learning for Free-Hand Sketch: A Survey (2020, methodology): “Current issues in SBIR include: Self-supervised pre-training for SBIR [115], [239], optimizing SBIR for early retrieval using partially drawn sketches, for example”