Sketch-based 3D shape retrieval using Convolutional Neural Networks

By Fang Wang, Le Kang, Yi Li. Computer Vision and Pattern Recognition, 2015

382 citations recorded by the source corpus.

Retrieving 3D models from 2D human sketches has received considerable attention in the areas of graphics, image retrieval, and computer vision. Almost always in state of the art approaches a large amount of “best views” are computed for 3D models, with the hope that the query sketch matches one of these 2D projections of 3D models using predefined features. We argue that this two stage approach (view selection - matching) is pragmatic but also problematic because the “best views” are subjective and ambiguous, which makes the matching inputs obscure. This imprecise nature of matching further makes it challenging to choose features manually. Instead of relying on the elusive concept of “best views” and the hand-crafted features, we propose to define our views using a minimalism approach and learn features for both sketches and views. Specifically, we drastically reduce the number of views to only two predefined directions for the whole dataset. Then, we learn two Siamese Convolutional Neural Networks (CNNs), one for the views and one for the sketches. The loss function is defined on the within-domain as well as the cross-domain similarities. Our experiments on three benchmark datasets demonstrate that our method is significantly better than state of the art approaches, and outperforms them in all conventional metrics.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Geology

View Paper: https://www.semanticscholar.org/paper/c43986083db2a17f10caf99cda6b227048fa58b7

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Geology

Citations in the corpus

SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis (2018, background): “There exist numerous works on sketch-based image retrieval [11, 12, 21, 2, 3, 55, 23, 22, 26, 54, 38, 56, 34].”
Triplet-Center Loss for Multi-view 3D Object Retrieval (2018, methodology): “5, a comprehensive comparison with various statethe-of-arts is presented, including Sketch-Based Retrieval method with View Clustering (SBR-VC) [17], Cross Domain Manifold Ranking method (CDMR) [10], Siamese network (Siamese) [35], Deep Correlated Metric Learning (DCML) [9] and Learned Wasserstein B”
Cross-View Image Matching for Geo-Localization in Urban Environments (2017, methodology): “The Siamese network [4] has been utilized in image matching [11, 26], tracking [20] and retrieval [23].”
Zero-Shot Sketch-Image Hashing (2018, methodology): “Siamese CNN [49], SaN [69], GN Triplet [52], 3D Shape [61] and DSH [39] are involved as SBIR baselines.”
Learning Weight Uncertainty with Stochastic Gradient MCMC for Shape Classification (2016, background): “Interestingly, due to the intrinsic complexity of shape geometry, the convolution operator has been applied in a variety of spaces, including multi-view projections [3, 66, 75], 3D volumes [79], curvature space [31] and meshed graphs [7].”
Cross-Modal Hierarchical Modelling for Fine-Grained Sketch Based Image Retrieval (2020, methodology): “Sketches as an alternative query mainly focussed on categorylevel retrieval at first [2, 6, 7, 44].”
Adapting Models to Signal Degradation using Distillation (2016, background): “…and staged training suggests that a better strategy to recognize line drawings of shapes used in various sketch-based retrieval applications [43, 51] is to ﬁrst ﬁne-tune the model on realistically rendered 3D models ( e.g . with shading and texture) then distill the model to edge images.”
LiveSketch: Query Perturbations for Guided Sketch-Based Visual Search (2019, background): “Wang et al [34] were arguably the ﬁrst to explore CNNs for sketched 3D model retrieval via a contrastive loss network mapping sketches to rendered 2D views.”
Learning in high-dimensional multimedia data: the state of the art (2015, methodology): “The outputs of feature reduction approaches are taken as the inputs for supervised, semi-supervised or unsupervised learning to support multimedia real applications such as multimedia retrieval, multimedia annotation and video tracking [19, 33, 77, 88].”
Doodle to Search: Practical Zero-Shot Sketch-Based Image Retrieval (2019, background): “Based on the problem at hand, two separated tasks have been identified: (1) Fine-grained SBIR (FGSBIR) aims to capture fine-grained similarities of sketch and photo [14, 25, 34] and (2) Coarse-grained SBIR (CGSBIR) performs a instance level search across multiple object categories [35, 9, 10, 29, 35”
Localizing and Orienting Street Views Using Overhead Imagery (2016, methodology): “The Siamese-like network, shown in Figure 3(b), has been used for cross-view image matching [20, 34] and retrieval [30, 4].”
Semantically Tied Paired Cycle Consistency for Zero-Shot Sketch-Based Image Retrieval (2019, methodology): “The SBIR methods that we evaluate are SaN [60], 3D Shape [49], Siamese CNN [36], GN Triplet [42], DSH [27] and GDH [63].”
Sketch-a-Net: A Deep Neural Network that Beats Humans (2016, methodology): “One exception is the sketchto-3D-shape retrieval work in Wang et al (2015).”
Towards 3D VR-Sketch to 3D Shape Retrieval (2020, methodology): “Various input modalities have been tried – images [14] and rough 2D sketch [32] at ﬁrst, with latest research focusing on 3D to 3D, i.e., using 3D scans [1, 5] or existing 3D shapes [30, 24, 21, 7, 9, 10, 12, 23, 25, 31].”
Learning Aligned Cross-Modal Representations from Weakly Aligned Data (2016, neutral): “[8] focuses on recovering semantically related natural images to a given sketch query and [41] uses query sketches to recover 3D shapes.”
Deep Sketch Hashing: Fast Free-Hand Sketch-Based Image Retrieval (2017, methodology): “Unlike previous methods with Siamese [43, 57] or triplet CNNs [49, 62] only utilizing images and sketches, we propose a novel semi-heterogeneous deep architecture including three CNNs, where a unique middle-level network fed with “sketch-tokens” is developed to effectively diminish the aforementione”
Semantic-Aware Knowledge Preservation for Zero-Shot Sketch-Based Image Retrieval (2019, background): “opment of deep learning, researchers have introduced deep neural networks into this field [22, 45, 21, 43, 37, 33, 30, 44, 42, 39].”
Deep Metric Learning: A Survey (2019, background): “Both sketch and 3D shape of images benefited for 3D shape retrieval while using shared weights and a metric loss function in [23,45,47].”