Learning cross-modality similarity for multinomial data

By Yangqing Jia, Mathieu Salzmann, Trevor Darrell. 2011

203 citations recorded by the source corpus.

Many applications involve multiple-modalities such as text and images that describe the problem of interest. In order to leverage the information present in all the modalities, one must model the relationships between them. While some techniques have been proposed to tackle this problem, they either are restricted to words describing visual objects only, or require full correspondences between the different modalities. As a consequence, they are unable to tackle more realistic scenarios where a narrative text is only loosely related to an image, and where only a few image-text pairs are available. In this paper, we propose a model that addresses both these challenges. Our model can be seen as a Markov random field of topic models, which connects the documents based on their similarity. As a consequence, the topics learned with our model are shared across connected documents, thus encoding the relations between different modalities. We demonstrate the effectiveness of our model for image retrieval from a loosely related text.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Visual Features from Large Weakly Supervised Data (2015, neutral): “Our work is also related to prior studies on multi-modal embedding [44, 56] that explore approaches such as kernel canonical component analysis [16, 18], restricted Boltzmann machines [45], topic models [22], and log-bilinear models [25].”
Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models (2014, background): “as. Other approaches to multimodal learning include the use of deep Boltzmann machines [16], log-bilinear neural language models [2], autoencoders [17], recurrent neural networks [7] and topic-models [18]. Several bi-directional approaches to ranking images and captions have also been proposed, base”
Deep Fragment Embeddings for Bidirectional Image Sentence Mapping (2014, methodology): “Several probabilistic models for representing joint multimodal probability distributions over images and sentences have been developed, using Deep Boltzmann Machines [16], log-bilinear models [17], and topic models [18, 19].”
Deep Visual-Semantic Alignments for Generating Image Descriptions (2014, neutral): “We then treat these correspondences as training data for our multimodal Recurrent Neural Network model which learns to generate the descriptions.”
Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN) (2014, methodology): “Accordingly, when we do backpropagation, we need to propagate the error through recurrent connections back in time (Rumelhart et al. (1988)).”
Explain Images with Multimodal Recurrent Neural Networks (2014, methodology): “sentences and images), using for example, Deep Boltzmann Machines [35], and topic models [1, 15].”
DenseCap: Fully Convolutional Localization Networks for Dense Captioning (2015, background): “Several pioneering approaches have explored the task of describing images with natural language [1, 27, 12, 34, 42, 43, 28, 20].”
Jointly Modeling Embedding and Translation to Bridge Video and Language (2015, background): “In this direction, several works explore such probability distribution using topic models [3, 13] and neural networks [6, 16, 20, 31, 32, 33, 35].”
Joint Video and Text Parsing for Understanding Events and Answering Queries (2013, background): “…Liu et al. [18] presented an image retrieval system by leveraging large-scale web image and their associated textual descriptions; Jia et al. [19] proposed a probabilistic model that encodes relations between loosely related images and text descriptions; Feng and Lapata [20] learned a…”
Domain Adaptation for Visual Applications: A Comprehensive Survey (2017, background): “ation analyses [110], co-training [34], spectral embedding [203], multiple kernel learning [71]. Multi-view intermediate domains are also often exploited to deﬁne for example cross-modal similarities [3, 121], semantic [246, 174, 87, 256] or multi-view embedding [110, 153, 196, 104, 25, 238] especia”
Fluency-Guided Cross-Lingual Image Captioning (2017, background): “, [11, 18, 32] similarly rank existing descriptions but in a common multimodal space for the visual and textual data.”
Cross-Modal Retrieval With CNN Visual Features: A New Baseline (2017, background): “[20] presented a topic model to learn cross-modality similarity for multimodal data.”
Video Event Understanding Using Natural Language Descriptions (2013, background): “Another line of work [9, 29] jointly considers multiple modalities including text descriptions to perform image annotation, retrieval or segmentation.”
A Sentence Is Worth a Thousand Pixels (2013, background): “This is further extended in [12] so that each image does not have to be associated with a text description.”
A Comprehensive Survey on Cross-modal Retrieval (2016, methodology): “Corr-LDA [21], Tr-mm LDA [22], MDRF [23] Deep learning methods Multimodal Deep Autoencoder [24], Multimodal DBM [25], DCCA [26], End-to-end DCCA [27], Corr-AE [28], Joint Video-Language Model [29] Pairwise based methods”
Convolutional Image Captioning (2017, background): “Early techniques formulate image captioning as a retrieval problem and find the best fitting description from a pool of possible captions [11, 15, 22, 32].”
Learning Cross-Media Joint Representation With Sparse and Semisupervised Regularization (2014, background): “The first is to design a heterogeneous similarity measure, by which we could directly compute the similarity or ranking value between heterogeneous features without giving an explicit representation [11], [17]–[21].”