LiveSketch: Query Perturbations for Guided Sketch-Based Visual Search

By John Philip Collomosse, Tu Bui, Hailin Jin. 2019

62 citations recorded by the source corpus.

LiveSketch is a novel algorithm for searching large image collections using hand-sketched queries. LiveSketch tackles the inherent ambiguity of sketch search by creating visual suggestions that augment the query as it is drawn, making query specification an iterative rather than one-shot process that helps disambiguate users' search intent. Our technical contributions are: a triplet convnet architecture that incorporates an RNN based variational autoencoder to search for images using vector (stroke-based) queries; real-time clustering to identify likely search intents (and so, targets within the search embedding); and the use of backpropagation from those targets to perturb the input stroke sequence, so suggesting alterations to the query in order to guide the search. We show improvements in accuracy and time-to-task over contemporary baselines using a 67M image corpus.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketches (2022, methodology): “For learning the joint embedding space, category-level SBIR typically employs either CNN [11, 12], RNN [65], or Transformer [41] based Siamese networks, accompanied by a triplet-ranking objective [68].”
Sketch3T: Test-Time Training for Zero-Shot SBIR (2022, background): “Shifting to deep-learning, methods [11, 28, 64] usually trained Siamese-like networks to fetch similar photos over a distance-metric in a crossmodal joint embedding space, over ranking losses [12].”
Deep Learning for Free-Hand Sketch: A Survey (2020, background): “are ranking-based, including triplet ranking [209], [222], [223] and quadruplet ranking [224].”
Sketch Less for More: On-the-Fly Fine-Grained Sketch-Based Image Retrieval (2020, unknown): “(iii) In the context of on-the-fly FGSBIR where we possess online sketch-stroke information, a reasonable alternative could be using a recurrent neural network like [4] for modelling the sketch branch instead of CNN.”
Partially Does It: Towards Scene-Level FG-SBIR with Partial Input (2022, background): “The prevailing nature of touch-screen devices has triggered significant research progress on sketches [6, 17, 25, 37, 45].”
More Photos are All You Need: Semi-Supervised Learning for Fine-Grained Sketch Based Image Retrieval (2021, background): “Despite starting as a category-level retrieval problem [9, 10, 3, 12], the fine-grained nature of sketches stirred current research focus more towards fine-grained SBIR (FGSBIR) [5, 28] – which aims to retrieve a particular photo based on a query sketch at an intra-category basis.”
Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting (2021, methodology): “For RNN-like architectures [22], we feed the five elements vector vt at every time step, and take the hidden state of final time step as the encoded latent representation.”
Sketching without Worrying: Noise-Tolerant Sketch-Based Image Retrieval (2022, methodology): “Like any other retrieval system, Deep Neural Networks have become a de-facto choice for any recent SBIR frameworks [15, 13, 37, 60, 12, 7] over early hand-engineered feature descriptors [50].”
Cross-Modal Hierarchical Modelling for Fine-Grained Sketch Based Image Retrieval (2020, methodology): “Similar sketch-embedding topology has been recently used in SketchMate [48] and LiveSketch [7], where RNN embeds coordinate representation.”
StyleMeUp: Towards Style-Agnostic Sketch-Based Image Retrieval (2021, background): “Most recent approaches are based on deep learning [30, 8].”
Sketchformer: Transformer-Based Representation for Sketched Structure (2020, methodology): “3 visualizes the learned embedding for each of the three pre-processing variants, alongside that of a state of the art sketch encoding model using stroke sequences [5].”