Grounded Compositional Semantics for Finding and Describing Images with Sentences

By Richard Socher, Andrej Karpathy, Quoc V. Le, Christopher D. Manning, Andrew Y. Ng. 2014

904 citations recorded by the source corpus.

Previous work on Recursive Neural Networks (RNNs) shows that these models can produce compositional feature vectors for accurately representing and classifying sentences or images. However, the sentence vectors of previous models cannot accurately represent visually grounded meaning. We introduce the DT-RNN model which uses dependency trees to embed sentences into a vector space in order to retrieve images that are described by those sentences. Unlike previous RNN-based models which use constituency trees, DT-RNNs naturally focus on the action and agents in a sentence. They are better able to abstract from the details of word order and syntactic expression. DT-RNNs outperform other recursive and recurrent neural networks, kernelized CCA and a bag-of-words baseline on the tasks of finding an image that fits a sentence description and vice versa. They also give more similar representations to sentences that describe the same image.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Learned in Translation: Contextualized Word Vectors (2017, methodology): “This initialization improves accuracy on other related tasks such as visual question answering [Xiong et al., 2016] or image captioning [Lu et al., 2016, Socher et al., 2014].”
Skip-Thought Vectors (2015, neutral): “3822 SDT-RNN [23] 0.”
Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books (2015, methodology): “In [13, 34], the authors use RNN embeddings to find the correspondences.”
Learning Transferable Visual Models From Natural Language Supervision (2021, background): “Over time work explored many combinations of training objective, transfer, and more expressive models and steadily improved performance (Frome et al., 2013; Socher et al., 2014; Karpathy et al., 2014; Kiros et al., 2014; Faghri et al., 2017).”
A Neural Network for Factoid Question Answering over Paragraphs (2014, methodology): “To take advantage of this observation, we depart from Socher et al. (2014) by training both the answers and questions jointly in a single model, rather than training each separately and holding embeddings ﬁxed during dt-rnn training.”
A Convolutional Neural Network for Modelling Sentences (2014, background): “The ﬁrst two experiments involve predicting the sentiment of movie reviews (Socher et al., 2013b).”
Learning with Latent Language (2017, neutral): “For the example in Figure 2, f might be an image rating model (Socher et al., 2014) that outputs a scalar judgment y of how well an image xmatches a caption w.”
Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models (2014, methodology): “Popular approaches include learning joint image-word embeddings [4, 5] as well as embedding images and sentences into a common space [6, 15].”
Deep Fragment Embeddings for Bidirectional Image Sentence Mapping (2014, methodology): “[22] with a single fullframe CNN as “Socher et al”, and to their method with our combined CNN features as “SDT-RNN”.”
Combined Scaling for Zero-shot Transfer Learning (2021, neutral): “…et al., 2010; Socher and Fei-Fei, 2010; Socher et al., 2013; Hodosh et al., 2013; Frome et al., 2013; Norouzi et al., 2013; Kiros et al., 2014; Socher et al., 2014; Akata et al., 2015b,a; Nam et al., 2017; Faghri et al., 2017; Li et al., 2019; Liu et al., 2019; Lu et al., 2019; Messina et…”
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision (2021, methodology): “…(Frome et al., 2013; Faghri et al., 2018) and improved versions (e.g., leveraging object detectors, dense feature maps, or multi-attention layers) (Socher et al., 2014; Karpathy et al., 2014; Kiros et al.; Nam et al., 2017; Li et al., 2019; Messina et al., 2020; Chen et al., 2020a) have been…”
Long-term recurrent convolutional networks for visual recognition and description (2014, methodology): “We first show the effectiveness of our model by quantitatively evaluating it on the image and caption retrieval tasks proposed by [26] and seen in [25, 15, 36, 8, 18].”
Show and tell: A neural image caption generator (2014, background): “For this purpose, it is instructive to think of the LSTM in un-rolled form – a copy of the LSTM memory is created for the image and each sentence word such that all LSTMs share the same parameters and the output m t − 1 of the LSTM at time t − 1 is fed to the LSTM at time t (see Figure 3).”
Deep Visual-Semantic Alignments for Generating Image Descriptions (2014, neutral): “For MSCOCO we use 5,000 images for both validation and testing.”
Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks (2015, methodology): “Our results are summarized in (Socher et al., 2014) 0.7863 0.7305 0.3983 SDT-RNN (Socher et al., 2014) 0.7886 0.7280 0.3859 Illinois-LH (Lai and Hockenmaier, 2014) 0.7993 0.7538 0.3692 UNAL-NLP (Jimenez et al., 2014) 0.8070 0.7489 0.3550 Meaning Factory (Bjerva et al., 2014) 0.8268 0.7721 0.3224…”
Learning Deep Structure-Preserving Image-Text Embeddings (2015, methodology): “As a more powerful objective function, a few other works have proposed a bi-directional ranking loss that, in addition to ensuring that correct sentences for each training image get ranked above incorrect ones, also ensures that for each sentence, the image described by that sentence gets ranked abo”
Visual7W: Grounded Question Answering in Images (2015, background): “Pioneer work in image captioning [4, 5, 13, 45, 47], sentence-based image retrieval [14, 40] and visual QA [1, 7, 36] shows promising results.”
Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding (2016, background): “, 2015), linear models with ranking loss (Frome et al., 2013; Karpathy and Fei-Fei, 2015; Socher et al., 2014; Weston et al., 2011) or non-linear deep learning models (Kiros et al.”
From captions to visual concepts and back (2014, background): “Recent retrievalbased approaches have used neural networks to map images and text into a common vector representation [11, 42, 40].”