Multimodal Neural Language Models

By Ryan Kiros, Ruslan Salakhutdinov, Richard S. Zemel. 2014

708 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Multimodal Learning

Citations in the corpus

Learning Visual Features from Large Weakly Supervised Data (2015, neutral): “Our work is also related to prior studies on multi-modal embedding [44, 56] that explore approaches such as kernel canonical component analysis [16, 18], restricted Boltzmann machines [45], topic models [22], and log-bilinear models [25].”
Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models (2014, methodology): “We show that sampling from an SC-NLM allows us to generate realistic image captions, signiﬁcantly improving over the generated captions produced by [2].”
Deep Fragment Embeddings for Bidirectional Image Sentence Mapping (2014, methodology): “l data. Several probabilistic models for representing joint multimodal probability distributions over images and sentences have been developed, using Deep Boltzmann Machines [15], log-bilinear models [16], and topic models [17, 18]. Ngiam et al. [19] described an autoencoder that learns audio-video ”
Conditional Generative Adversarial Nets (2014, background): “Additionally, in [13] the authors show how to train a supervised multi-modal neural language model, and they are able to generate descriptive sentence for images.”
VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, background): “With the advent of deep learning, researchers proposed end-to-end neural networks that encode an image into vector representations and decode a caption word by word [36, 82, 21, 32].”
Microsoft COCO Captions: Data Collection and Evaluation Server (2015, background): “Recently there has been a surprising resurgence of interest in this area [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [30], due to the renewed interest in neural network learning techniques [31], [32] and increasingly large datasets [33], [34], [35], [7], [36], [37], [38].”
Show and tell: A neural image caption generator (2014, background): “All LSTMs share the same parameters. image and each sentence word such that all LSTMs share the same parameters and the output mt−1 of the LSTM at time t − 1 is fed to the LSTM at time t (see Figure 3).”
Deep Visual-Semantic Alignments for Generating Image Descriptions (2014, neutral): “hat are ﬁlled based on the content of the image [19,29,13,55,56,9,1] or generative grammars [42,57], but this approach limits the variety of possible outputs. Most closely related to us, Kiros et al. [26] developed a logbilinear model that can generate full sentence descriptions for images, but thei”
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention (2015, methodology): “(2014), Karpathy & Li (2014) and Kiros et al. (2014b). For fairness, we only compare against results for which we have verified that our BLEU evaluation code is the same.”
Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models (2015, background): “…machine learning tasks including statistical machine translation (Auli et al. 2013; Sutskever, Vinyals, and Le 2014; Bahdanau, Cho, and Bengio 2015), speech recognition (Hinton et al. 2012; Deng and Li 2013) and image caption generation (Kiros, Salakhutdinov, and Zemel 2014; Vinyals et al. 2015).”
Learning Deep Structure-Preserving Image-Text Embeddings (2015, background): “Another strand of research on multi-modal embeddings is based on deep learning [3, 24, 25, 31, 35, 44], utilizing such techniques as deep Boltzmann machines [44], autoencoders [35], LSTMs [8], and recurrent neural networks [31, 45].”
Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding (2016, background): “…et al., 2004) e.g. (Gong et al., 2014; Klein et al., 2015; Plummer et al., 2015), linear models with ranking loss (Frome et al., 2013; Karpathy and Fei-Fei, 2015; Socher et al., 2014; Weston et al., 2011) or non-linear deep learning models (Kiros et al., 2014; Mao et al., 2015; Ngiam et al., 2011).”
From captions to visual concepts and back (2014, methodology): “[18] use a log-bilinear model with bias features derived from the image to model text conditional on the image.”
Generating Images from Captions with Attention (2015, background): “One approach is to learn a generative model of text conditioned on images, known as caption generation (Kiros et al., 2014a; Karpathy & Li, 2015; Vinyals et al., 2015; Xu et al., 2015).”
Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN) (2014, methodology): “For the sentence retrieval task, Kiros et al. (2014b) uses a shortlist of 100 images which are the nearest neighbors of the query image in the feature space.”
Movie Description (2016, background): “Recently image description has gained increased attention with work such as that of Chen and Zitnick (2015); Donahue et al. (2015); Fang et al. (2015); Karpathy and Fei-Fei (2015); Kiros et al. (2014, 2015); Mao et al. (2015); Vinyals et al. (2015); Xu et al. (2015a).”
A dataset for Movie Description (2015, background): “In recent years there has been an increased interest in automatically describing images [23, 39, 40, 50, 45, 40, 41, 34, 61, 22] and videos [37, 27, 8, 28, 32, 62, 16, 26, 64, 55] with natural language.”