Shared Predictive Cross-Modal Deep Quantization

By Erkun Yang, Cheng Deng, Chao Li, Wei David Liu, Jie Li, Dacheng Tao. 2018

137 citations recorded by the source corpus.

With explosive growth of data volume and ever-increasing diversity of data modalities, cross-modal similarity search, which conducts nearest neighbor search across different modalities, has been attracting increasing interest. This paper presents a deep compact code learning solution for efficient cross-modal similarity search. Many recent studies have proven that quantization-based approaches perform generally better than hashing-based approaches on single-modal similarity search. In this paper, we propose a deep quantization approach, which is among the early attempts of leveraging deep neural networks into quantization-based cross-modal similarity search. Our approach, dubbed shared predictive deep quantization (SPDQ), explicitly formulates a shared subspace across different modalities and two private subspaces for individual modalities, and representations in the shared subspace and the private subspaces are learned simultaneously by embedding them to a reproducing kernel Hilbert space, where the mean embedding of different modality distributions can be explicitly compared. In addition, in the shared subspace, a quantizer is learned to produce the semantics preserving compact codes with the help of label alignment. Thanks to this novel network architecture in cooperation with supervised quantization training, SPDQ can preserve intramodal and intermodal similarities as much as possible and greatly reduce quantization error. Experiments on two popular benchmarks corroborate that our approach outperforms state-of-the-art methods.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Qi Tian
3. Cheng Deng
4. Yi Yang
5. Tongliang Liu
6. Jun Yu
7. Fei Wang
8. Chen Qian

Citations in the corpus

• GreedyNAS: Towards Fast One-Shot NAS With Greedy Supernet (2020, background): “By dint of automatic feature engineering, deep neural networks (DNNs) have achieved remarkable success in various computer vision tasks, such as image classification [37, 36, 41, 33, 32, 15, 39], visual generation [34, 35], image retrieval [40, 7, 8, 12] and semantic comprehension [18, 17].”
• Class-Incremental Instance Segmentation via Multi-Teacher Networks (2021, background): “Due to the development of deep learning, like other visual tasks (Yang et al. 2018, 2020a; Dang et al. 2020; Yang et al. 2020b; Deng et al. 2019), instance segmentation meth-ods have achieved amazing results.”
• Two-Stage Label Embedding via Neural Factorization Machine for Multi-Label Classification (2019, methodology): “Embedding method (Hsu et al. 2009; Cao et al. 2016; Yang et al. 2018; Deng et al. 2018a) is one of the most popular frameworks to learn label and feature correlations and has shown promising results.”
• Two-Stream Deep Hashing With Class-Specific Centers for Supervised Image Search (2020, background): “S IMILARITY search is a fundamental research topic in numerous computer vision and multimedia applications, including object recognition [1], information retrieval [2]–[19], image classification [20], [21], motion estimation [22], and image understanding [23], [24].”
• COMIC: Multi-view Clustering Without Parameter Selection (2019, background): “Kumar et al. (2011); Xu et al. (2015); Zhang et al. (2017); Yang et al. (2018) have proven that simply applying single-view clustering methods cannot narrow the heterogeneous gap to achieve desirable results because the clusters may largely differ in each of the data views.”
• Multimodal Unified Attention Networks for Vision-and-Language Interactions (2019, background): “Such advances drive research interest in multimodal learning tasks lying at the intersection of vision and language such as multimodal embedding learning [1][2][3], visual captioning [4][5], visual question answering (VQA) [6] and visual grounding [7], etc.”
• Agree to Disagree: Adaptive Ensemble Knowledge Distillation in Gradient Space (2020, background): “Recently, deploying deep neural networks on edge devices has appealed much attention due to their achieved success in various tasks [32, 19, 36, 34, 13, 35, 33, 15, 38].”