Discriminative coupled dictionary hashing for fast cross-media retrieval

By Zhou Yu, Fei Wu, Yi Yang, Qi Tian, Jiebo Luo, Yueting Zhuang. 2014

150 citations recorded by the source corpus.

Cross-media hashing, which conducts cross-media retrieval by embedding data from different modalities into a common low-dimensional Hamming space, has attracted intensive attention in recent years. The existing cross-media hashing approaches only aim at learning hash functions to preserve the intra-modality and inter-modality correlations, but do not directly capture the underlying semantic information of the multi-modal data. We propose a discriminative coupled dictionary hashing (DCDH) method in this paper. In DCDH, the coupled dictionary for each modality is learned with side information (e.g., categories). As a result, the coupled dictionaries not only preserve the intra-similarity and inter-correlation among multi-modal data, but also contain dictionary atoms that are semantically discriminative (i.e., the data from the same category is reconstructed by the similar dictionary atoms). To perform fast cross-media retrieval, we learn hash functions which map data from the dictionary space to a low-dimensional Hamming space. Besides, we conjecture that a balanced representation is crucial in cross-media retrieval. We introduce multi-view features on the relatively ``weak'' modalities into DCDH and extend it to multi-view DCDH (MV-DCDH) in order to enhance their representation capability. The experiments on two real-world data sets show that our DCDH and MV-DCDH outperform the state-of-the-art methods significantly on cross-media retrieval.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Multi-modal Factorized Bilinear Pooling with Co-attention Learning for Visual Question Answering (2017, background): “Such advances have ∗ Jun Yu is the corresponding author also stimulated new research topics like image-text retrieval [35, 37], image captioning [5, 34], and visual question answering [3, 19].”
Joint Dictionaries for Zero-Shot Learning (2017, background): “The intuition from a co-view perspective (Yu et al. 2014) is that both the visual and the attribute features provide information about the same class, and so each can augment the learning of the other.”
Multimodal Transformer With Multi-View Visual Representation for Image Captioning (2019, background): “2947482 multimodal learning tasks such as image-text matching [1],”
An Overview of Cross-Media Retrieval: Concepts, Methodologies, Benchmarks, and Challenges (2017, background): “…[56](U) [57](F) [58](F) [59](F) Dictionary learning [61](U) [62](S,F) [64](F) Cross-mediahashing [67](U) [68](U) [49](F) [69](U) [70](F) [71](F) [72](F) [73](U) [74](F) [75](U) [76](U) [77](F) [78](F) [79](U) [80](U) [81](F) [82](U) [83](U) Others [20](U) [84](F) [85](F) [86](F) [87](F) [89](S)”
Transfer Knowledge between Cities (2016, methodology): “Though the works [8, 31] also learn dictionaries by graph clustering, they either learn a dictionary for single modality, or ignore unlabeled vertices and modality diversity in each cluster for multi-modalities.”
Cross-Modal Retrieval With CNN Visual Features: A New Baseline (2017, background): “To address the problem of prohibitively expensive nearest neighbor search, some hashingbased approaches [3], [26], [28], [43], [44], [59], [65], [67], [69], [70] to large scale similarity search have drawn much interest from the cross-modal retrieval community.”
Semantics-preserving hashing for cross-view retrieval (2015, methodology): “Different from unsupervised cross-view hashing meth-ods, supervised ones [1, 9, 24, 23, 18, 17, 26, 19] are proposed to further exploit available supervised information like labels or semantic afﬁnities of training data for improving performance.”
Using Task Descriptions in Lifelong Machine Learning for Improved Performance and Zero-Shot Transfer (2017, background): “…coupled dictionary learning has led to high performance algorithms for image super-resolution (Yang et al., 2010), allowing the reconstruction of high-res images from low-res samples, and for multi-modal retrieval (Zhuang, Wang, Wu, Zhang, & Lu, 2013) and cross-domain retrieval (Yu et al., 2014).”
ROSITA: Enhancing Vision-and-Language Semantic Alignments via Cross- and Intra-modal Knowledge Integration (2021, unknown): “Given an image-text pair, we resort to a unified scene graph structure G =< V ,E, S > to encode its intra- and cross-modal knowledge simultaneously [48].”
Learning Transferable Knowledge Through Embedding Spaces (2019, background): “From a co-view perspective [65], both the policies and descriptors provide information about the task, and so each can augment the learning of the other.”
A Comprehensive Survey on Cross-modal Retrieval (2016, methodology): “Similarly, Yu et al. [79] propose a discriminative coupled dictionary hashing (DCDH) method to capture the underlying semantic information of the multi-modal data.”
Joint Feature Selection and Subspace Learning for Cross-Modal Retrieval (2016, background): “To guarantee that each class has abundant training samples like [57], we select those pairs that belong to one of the 21 largest classes with each pair exclusively belonging to one of the 21 classes, which results in 72,219”
A Survey of Multi-View Representation Learning (2016, unknown): “However, CMSSH only considers the inter-view correlation but ignores the intra-view similarity [157].”
Multimodal Unified Attention Networks for Vision-and-Language Interactions (2019, background): “Such advances drive research interest in multimodal learning tasks lying at the intersection of vision and language such as multimodal embedding learning [1][2][3], visual captioning [4][5], visual question answering (VQA) [6] and visual grounding [7], etc.”