A new approach to cross-modal multimedia retrieval

By Nikhil Rasiwasia, José Costa Pereira, Emanuele Coviello, Gabriel Doyle, Gert R. G. Lanckriet, Roger Levy, Nuno Vasconcelos. 2010

1,376 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Information Retrieval & Search, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Information Retrieval & Search

In Multimodal Learning

Citations in the corpus

Self-Supervised Learning of Visual Features through Embedding Images into Text Topic Spaces (2017, positive): “Table 5: MAP comparison on Wikipedia dataset [29] with supervised (bottom) and unsupervised (middle) methods.”
Achieving Human Parity on Visual Question Answering (2021, background): “For example, Sharma et al. [66] extend CCA to a supervised version by leveraging the the semantic category labels to guide the representation learning.”
A Multi-View Embedding Space for Modeling Internet Images, Tags, and Their Semantics (2012, methodology): “We have started with the two-view visual-textual CCA model popular in several recent works (Gong and Lazebnik, 2011; Hardoon et al., 2004; Hwang and Grauman, 2010, 2011; Rasiwasia et al., 2010) and shown that its performance can be significantly improved by adding a third view based on semantic grou”
Domain Adaptation for Visual Applications: A Comprehensive Survey (2017, background): “], co-training [34], spectral embedding [203], multiple kernel learning [71]. Multi-view intermediate domains are also often exploited to deﬁne for example cross-modal similarities [3, 121], semantic [246, 174, 87, 256] or multi-view embedding [110, 153, 196, 104, 25, 238] especially used to perform”
Quantifying the Visual Concreteness of Words and Topics in Multimodal Datasets (2018, background): “This task is a good representative of multimodal learning because computing a joint embedding of text and images is often a “ﬁrst step” for downstream tasks, e.g., cross-modal retrieval (Rasiwasia et al., 2010), image tagging (Chen et al., 2013), and caption generation (Kiros et al., 2015).”
Robust Unsupervised Cross-modal Hashing for Multimedia Retrieval (2020, background): “Fortunately, data described with multiple modalities are expected to share a certain space if they are related to similar topics [25]; thereby, we introduce two modal-specific projection matrices W1 ∈ Rd1×c and W2 ∈ Rd2×c to map the corresponding data into the shared space (with c dimensions) via W1”
Linear cross-modal hashing for efficient multimedia search (2013, neutral): “Keywords Cross-modal, hashing, index, multimedia search”
Dual-path Convolutional Image-Text Embeddings with Instance Loss (2017, background): “, Wikipedia [1] and Pascal Sentence [2], which”
On the Role of Correlation and Abstraction in Cross-Modal Multimedia Retrieval (2014, background): “Index Terms—Multimedia, content-based retrieval, multimodal, cross-modal, image and text, retrieval model, semantic spaces, kernel correlation, logistic regression Ç”
Cross-Modal Retrieval With CNN Visual Features: A New Baseline (2017, methodology): “[38] proposed a semantic correlation matching (SCM) approach, where the multiclass logistic regression is applied to the maximally correlated fea-”
Learning Discriminative Binary Codes for Large-scale Cross-modal Retrieval (2017, methodology): “[2] applied CCA to learn a maximally correlated subspace for cross-modal retrieval.”
Cross-Modal Similarity Learning: A Low Rank Bilinear Formulation (2014, methodology): “…be described as follows: S = { ( x i , x j ) | x i and x j are in the same class } , D = { ( x i , x j ) | x i and x j are not in the same class } , (1) where S is called as the must-link set containing the similar pairs and the D is called as the cannot-link set containing the dissimilar pairs.”
Multimodal Manifold Analysis by Simultaneous Diagonalization of Laplacians (2015, neutral): “However , many applications involve observations and measurements of data done using different modalities, such as multimedia documents [13], [14], [15], audio and video [16], [17], or medical imaging modalities like PET and CT [18].”
Cross-Modality Submodular Dictionary Learning for Information Retrieval (2014, methodology): “…CmSDLiscom-pared with the non-knowledge transfer method (PCA), correlation matching (CM) [ 31 ] , semantic matching (SM) [ 31 ] , semantic correlation matching (SCM) [ 7 ] , a bilinear model (BLM) [ 42 ] , generalized multiview linear discriminant analysis (GMLDA) [ 36 ]…”
Cross-modal Retrieval with Correspondence Autoencoder (2014, methodology): “Rasiwasia et al. [19] proposed correlation matching mapping the features of images and texts extracted by LDA into same representation space using CCA.”
Text-to-image Synthesis via Symmetrical Distillation Networks (2018, positive): “As a multimedia technology, cross-modal retrieval [6, 20, 29, 38, 43] has become a highlighted topic of research in the big data era.”
Cross-View Image Geolocalization (2013, methodology): “Our approach takes inspiration from recent works in cross-view data retrieval that tackle problems such as static cameras localization with satellite imagery [11], cross-view action recognition [16] and image-text retrieval [19, 23].”
Learning Consistent Feature Representation for Cross-Modal Multimedia Retrieval (2015, methodology): “Till now, many extensions and applications have been developed based on CCA [7], [10], [17], [30].”