Multimodal learning with deep Boltzmann machines

By Nitish Srivastava, Ruslan Salakhutdinov. 2012

1,812 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, background): “Srivastava & Salakhutdinov (2012) explored deep representation learning by training multimodal Deep Boltzmann Machines on top of low-level image and text tag feature features.”
A Retrieve-and-Edit Framework for Predicting Structured Outputs (2018, background): “A related idea is cross-modal embeddings, which jointly embed and align items in different domains (such as images and captions) using autoencoders [39, 2, 31, 10].”
Learning Visual Features from Large Weakly Supervised Data (2015, neutral): “Our work is also related to prior studies on multi-modal embedding [44, 56] that explore approaches such as kernel canonical component analysis [16, 18], restricted Boltzmann machines [45], topic models [22], and log-bilinear models [25].”
Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models (2014, methodology): “Other approaches to multimodal learning include the use of deep Boltzmann machines [16], log-bilinear neural language models [2], autoencoders [17], recurrent neural networks [7] and topic-models [18].”
Deep Fragment Embeddings for Bidirectional Image Sentence Mapping (2014, methodology): “Several probabilistic models for representing joint multimodal probability distributions over images and sentences have been developed, using Deep Boltzmann Machines [16], log-bilinear models [17], and topic models [18, 19].”
Grounded Compositional Semantics for Finding and Describing Images with Sentences (2014, background): “Among other recent work is that by Srivastava and Salakhutdinov (2012) who developed multimodal Deep Boltzmann Machines.”
Zero-Shot Learning Through Cross-Modal Transfer (2013, methodology): “e text corpora to learn semantic word representations. Their model does require a small amount of training data however for each class. Among other recent work is that by Srivastava and Salakhutdinov [31] who developed multimodal Deep Boltzmann Machines. Similar to their work, we use techniques from”
Generative Adversarial Text to Image Synthesis (2016, background): “Srivastava & Salakhutdinov (2012) developed a deep Boltzmann machine and jointly modeled images and text tags.”
Conditional Generative Adversarial Nets (2014, methodology): “[17] take a similar approach to this problem, and train a multi-modal Deep Boltzmann Machine on the MIR Flickr 25,000 dataset as we do in this work.”
GAN Inversion: A Survey (2021, neutral): “(DBMs) [154], [155], [156], [157], Variational Autoencoders (VAEs) [158], [159], Deep Autoregressive Models (DARs) [160], [161], [162], [163], [164], and Normalizing Flow Models (NFMs) [165], [166], [167], [168].”
Coupled Generative Adversarial Networks (2016, methodology): “When training data are given as tuples of corresponding images in different domains, several existing approaches [1, 2, 3, 4] can be applied.”
Multimodal Transformer for Unaligned Multimodal Language Sequences (2019, background): “Unlike learning multimodal representations from static domains such as image and textual attributes (Ngiam et al., 2011; Srivastava and Salakhutdinov, 2012), human language contains time-series and thus requires fusing time-varying signals (Liang et al., 2018; Tsai et al., 2019).”
Deep Visual-Semantic Alignments for Generating Image Descriptions (2014, neutral): “We then treat these correspondences as training data for our multimodal Recurrent Neural Network model which learns to generate the descriptions.”
Representation Learning: A Review and New Perspectives (2012, background): “The 1000-class ImageNet benchmark, whose results are detailed here: http://www.image-net.org/challenges/LSVRC/2012/results.html 5. downloadable from http://ml.nec-labs.com/senna/ 2010) and it has recently been extended to deeper multi-modal representations (Srivastava and Salakhutdinov, 2012).”
Learning to See by Moving (2015, background): “Another line of work has focused on learning features that are invariant to transformations irrelevant to the identity of objects, either from images [14, 34], or from video [41].”
Generative Image Modeling Using Spatial LSTMs (2015, methodology): “, 38, 39], but have also been used to model the progression of frames in video [36] and very recently to model single images [11].”
Least Squares Generative Adversarial Networks (2016, methodology): “ models attempt to capture the probability distributions over the given data samples. Deep generative models have been applied to data generation [7], image classication [12] and multimodal learning [13]. Restricted Boltzmann Machines (RBMs) are the basis of many other hierarchical models. RBMs have”
Learning Deep Structure-Preserving Image-Text Embeddings (2015, background): “Another strand of research on multi-modal embeddings is based on deep learning [3, 24, 25, 31, 35, 44], utilizing such techniques as deep Boltzmann machines [44], autoencoders [35], LSTMs [8], and recurrent neural networks [31, 45].”
From captions to visual concepts and back (2014, background): “Recent retrievalbased approaches have used neural networks to map images and text into a common vector representation [11, 42, 40].”
Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN) (2014, methodology): “The third category of methods, which is more related to our method, learns a probability density over the space of multimodal inputs (i.e. sentences and images), using for example, Deep Boltzmann Machines (Srivastava & Salakhutdinov (2012)), and topic models (Barnard et al. (2003); Jia et al. (2011)”