GIFT: A Real-Time and Scalable 3D Shape Search Engine

By Song Bai, Xiang Bai, Zhichao Zhou, Zhaoxiang Zhang, Longin Jan Latecki. Computer Vision and Pattern Recognition, 2016

294 citations recorded by the source corpus.

Projective analysis is an important solution for 3D shape retrieval, since human visual perceptions of 3D shapes rely on various 2D observations from different view points. Although multiple informative and discriminative views are utilized, most projection-based retrieval systems suffer from heavy computational cost, thus cannot satisfy the basic requirement of scalability for search engines. In this paper, we present a real-time 3D shape search engine based on the projective images of 3D shapes. The real-time property of our search engine results from the following aspects: (1) efficient projection and view feature extraction using GPU acceleration, (2) the first inverted file, referred as F-IF, is utilized to speed up the procedure of multi-view matching, (3) the second inverted file (S-IF), which captures a local distribution of 3D shapes in the feature manifold, is adopted for efficient context-based reranking. As a result, for each query the retrieval task can be finished within one second despite the necessary cost of IO overhead. We name the proposed 3D shape search engine, which combines GPU acceleration and Inverted File (Twice), as GIFT. Besides its high efficiency, GIFT also outperforms the state-of-the-art methods significantly in retrieval accuracy on various shape benchmarks and competitions.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/35ab37a5305b3db6a25b134667cb5829d51dbbec

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

SPLATNet: Sparse Lattice Networks for Point Cloud Processing (2018, background): “Multi-view networks pre-process shapes into a set of 2D rendered images encoding surface depth and normals under various 2D projections [39, 32, 3, 24, 9, 20].”
Deep Learning Advances in Computer Vision with 3D Data (2017, methodology): “A real-time 3D shape search engine based on 2D views of 3D objects was presented in Bai et al. [2016].”
Review: deep learning on 3D point clouds (2020, methodology): “These methods [17, 18, 19, 10, 20, 21, 22], take advantage of the already matured 2D CNNs into 3D.”
3D object classification and retrieval with Spherical CNNs (2017, methodology): “The top ranked teams either used pre-trained 2D-CNN on rendered views [30, 2], or handcrafted features [11], all with several million parameters.”
Learning SO(3) Equivariant Representations with Spherical CNNs (2017, background): “Currently, the most successful approaches are view-based, operating in rendered views of the 3D object [9,8,30,31].”
Deep Learning 3D Shapes Using Alt-az Anisotropic 2-Sphere Convolution (2018, methodology): “(2015); Bai et al. (2016),) or projecting a 3D object onto geometric entities which can be flattened as 2D images (Shi et al. (2015); Cao et al. (2017); Sfikas et al. (2018).) All these methods convert a 3D shape into an Euclidean grid structure which supports shift (translational) equivariance/inva”
MVTN: Multi-View Transformation Network for 3D Shape Recognition (2020, unknown): “Voxels 40.9 - 3D ShapeNets [34] Voxels 49.2 - PVNet[36] Points 89.5 - MVCNN [28] 12 Views 80.2 73.5 GIFT [1] 20 Views - 64.0 MVFusionNet [12] 12 Views - 62.2 ReVGG [27] 20 Views - 74.9 RotNet [14] 20 Views - 77.2 ViewGCN [32] 20 Views - 78.4 MLVCNN [13] 24 Views 92.2 - MVTN (ours) 12 Views 92.9…”
O-CNN (2017, methodology): “Multiple-view based approaches [Bai et al. 2016; Shi et al. 2015; Su et al. 2015] render the 3D shape into a set of 2D images observed from different views and feed the stacked images to the CNN.”
View-GCN: View-Based Graph Convolutional Network for 3D Shape Analysis (2020, unknown): “…current state-of-the-art method Ro-tationNet on this dataset on all metrics except micro-averaged NDCG. Compared with other methods such as GIFT [2], Improved-GIFT and MVCNN [47], view-GCN also achieves signiﬁcantly higher accuracies, e.g ., it achieves 4.9% (in mAP for microALL) higher than…”
Multi-view Harmonized Bilinear Network for 3D Object Recognition (2018, methodology): “In spite of that the view-base methods [29, 1, 15, 32] Figure 1.”
General-Purpose Deep Point Cloud Feature Extractor (2018, background): “Another avenue researchers frequently choose is to map the 3D data to 2D, either by creating images from multiple angles [48, 5, 24, 60], or by creating a mapping function from mesh to image such that depth information is preserved [44, 47].”
Pairwise Decomposition of Image Sequences for Active Multi-view Recognition (2016, unknown): “…Rather than modelling an object as a set of views with 2D features, an explicit 3D shape can be learned from reconstruction [37] or provided by CAD models [39], and subsequently matched to from depth images [13], 3D reconstructions [1], or partial reconstructions with shape completion [12, 39].”
SPNet: Deep 3D Object Classification and Retrieval using Stereographic Projection (2018, methodology): “56% GIFT [3] - 92.”
Triplet-Center Loss for Multi-view 3D Object Retrieval (2018, methodology): “View-based methods [1, 33] extract or learn the shape features from a set of 2D view projections, where 2D convolutional neural networks (CNN) are often adopted to process such projection images.”
Spherical Fractal Convolutional Neural Networks for Point Cloud Recognition (2019, unknown): “Zhou [2] 0.”
RotationNet: Joint Object Categorization and Pose Estimation Using Multiviews from Unsupervised Viewpoints (2016, methodology): “Multi-view representations are also used for 3D shape retrieval [1].”