Deep correlation for matching images and text

By Fei Yan, Krystian Mikolajczyk. 2015

410 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

A Retrieve-and-Edit Framework for Predicting Structured Outputs (2018, background): “A related idea is cross-modal embeddings, which jointly embed and align items in different domains (such as images and captions) using autoencoders [39, 2, 31, 10].”
Webly Supervised Joint Embedding for Cross-Modal Image-Text Retrieval (2018, methodology): “A number of image-text embedding approaches has been developed based on Canonical Correlation Analysis (CCA) [12, 16, 47, 57].”
Learning Deep Structure-Preserving Image-Text Embeddings (2015, methodology): “(a) State of the art Deep CCA [33] 27.”
MUTAN: Multimodal Tucker Fusion for Visual Question Answering (2017, background): “The main task in multimodal visual and textual analysis aims at learning an alignment between feature spaces [29, 23, 18].”
Dual Attention Networks for Multimodal Reasoning and Matching (2016, methodology): “We report the performance of mDAN in bidirectional image and sentence retrieval using the same metrics as previous work [34, 19, 20, 30].”
Multimodal Convolutional Neural Networks for Matching Image and Sentence (2015, methodology): “.0 47.7 12 12.8 32.9 44.5 13 RVP (T+I) [3] 12.1 27.8 47.8 11 12.7 33.1 44.9 12.5 DVSA (DepTree) [18] 20.0 46.6 59.4 5.4 15.0 36.5 48.2 10.4 DVSA (BRNN) [18] 22.2 48.2 61.4 4.8 15.2 37.7 50.5 9.2 DCCA [44] 16.7 39.3 52.9 8 12.6 31.0 43.0 15 NIC [42] 17.0 * 56.0 7 17.0 * 57.0 7 LRCN [6] * * * * 17.5 4”
Learning Two-Branch Neural Networks for Image-Text Matching Tasks (2017, background): “[25] and Yan and Mikolajczyk [26] proposed to cast CCA into a deep learning framework.”
Learning Joint Embedding with Multimodal Cues for Cross-Modal Video-Text Retrieval (2018, methodology): “Canonical Correlation Analysis (CCA) was used in several previous works for learning joint embeddings for the retrieval task [10, 13, 29, 37].”
Domain Adaptation for Visual Applications: A Comprehensive Survey (2017, background): “he target only images, the intermediate domain can be composed by a set of crawled Web pages containing both text and images. (Image courtesy B. Tan [222]). web pages or photos with tags or comments) [110, 210, 256] or images acquired with depth information [115]. We can also have multi-view situati”
Multi-Modality Cross Attention Network for Image and Sentence Matching (2020, methodology): “For the cross-modality association, the simplest way is to learn projection functions to map visual and textual data into the same embedding space, such as, canonical correlation objective [19, 50], structured objective [7, 26, 9, 52, 40].”
On Deep Multi-View Representation Learning (2015, background): “video (Kidron et al., 2005; Chaudhuri et al., 2009), audio + articulation (Arora and Livescu, 2013; Wang et al., 2015a), images + text (Hardoon et al., 2004; Socher and Li, 2010; Hodosh et al., 2013; Yan and Mikolajczyk, 2015), or parallel text in two languages (Vinokourov et al., 2003; Haghighi et ”
Learning Social Image Embedding with Deep Multimodal Attention Networks (2017, methodology): “On all the three datasets, DMAN outperforms KCCA and DCCA greatly.”
An Overview of Cross-Media Retrieval: Concepts, Methodologies, Benchmarks, and Challenges (2017, methodology): “as deep canonical correlation analysis (DCCA) [24], [35].”
Deep Multimodal Representation Learning: A Survey (2019, methodology): “Due to these advantages, DCCA has been widely used for various multi-view and multimodal learning tasks including word embedding in a multilingual context [124], [125], acoustic features representation [121], matching images and text [29], music retrieval [126], and speech recognition [127], [128].”
A Comprehensive Survey on Cross-modal Retrieval (2016, methodology): “], CFA [17], MCU [18], CoCA [19], MMD [20] Topic model Corr-LDA [21], Tr-mm LDA [22], MDRF [23] Deep learning methods Multimodal Deep Autoencoder [24], Multimodal DBM [25], DCCA [26], End-to-end DCCA [27], Corr-AE [28], Joint Video-Language Model [29] Pairwise based methods Shallow methods Multi-NPP”
DSSL: Deep Surroundings-person Separation Learning for Text-based Person Retrieval (2021, background): “As text-based person retrieval involves processing multi-modal data, it can be deemed as a specific subtask of cross-modal retrieval [9, 10, 15, 16, 21, 28].”
RNN Fisher Vectors for Action Recognition and Image Annotation (2015, background): “Action Recognition As in other object recognition problems, the standard pipeline in action recognition is comprised of three main steps: feature extraction, pooling and classification.”