Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketches

By Ayan Kumar Bhunia, Viswanatha Reddy Gajjala, Subhadeep Koley, Rohit Kundu, Aneeshan Sain, Tao A. Xiang, Yi-Zhe Song. 2022

33 citations recorded by the source corpus.

The human visual system is remarkable in learning new visual concepts from just a few examples. This is precisely the goal behind few-shot class incremental learning (FS-CIL), where the emphasis is additionally placed on ensuring the model does not suffer from “forgetting”. In this paper, we push the boundary further for FSCIL by addressing two key questions that bottleneck its ubiquitous application (i) can the model learn from diverse modalities other than just photo (as humans do), and (ii) what if photos are not readily accessible (due to ethical and privacy constraints). Our key innovation lies in advocating the use of sketches as a new modality for class support. The product is a “Doodle It Yourself” (DIY) FSCIL framework where the users can freely sketch a few examples of a novel class for the model to learn to recognise photos of that class. For that, we present a framework that infuses (i) gradient consensus for domain invariant learning, (ii) knowledge distillation for preserving old class information, and (iii) graph attention networks for message passing between old and novel classes. We experimentally show that sketches are better class support than text in the context of FSCIL, echoing findings elsewhere in the sketching literature.

Subfields: Computer Vision and Pattern Recognition, Education, Information Systems, Sociology and Political Science, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Education

In Information Systems

In Sociology and Political Science

Jun Zhu

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “Besides, auxiliary information such as semantic word vectors [262], [263] and sketch exemplars [264] can be incorporated to enrich the limited training samples.”
Sketching without Worrying: Noise-Tolerant Sketch-Based Image Retrieval (2022, methodology): “Like any other retrieval system, Deep Neural Networks have become a de-facto choice for any recent SBIR frameworks [15, 13, 37, 60, 12, 7] over early hand-engineered feature descriptors [50].”
Partially Does It: Towards Scene-Level FG-SBIR with Partial Input (2022, background): “Related Work Fine-grained sketch based image retrieval The ability of sketches to offer inherently fine-grained visual descriptions commences the avenues of fine-grained sketch-based image retrieval (FG-SBIR) [4, 7, 8, 53, 54, 58, 63].”
Sketch3T: Test-Time Training for Zero-Shot SBIR (2022, background): “Despite great strides made, the field remains plagued by the data scarcity problem – sketches are notoriously difficult to collect [4,6].”