Deep Fragment Embeddings for Bidirectional Image Sentence Mapping

By Andrej Karpathy, Armand Joulin, Li Fei-Fei. 2014

980 citations recorded by the source corpus.

We introduce a model for bidirectional retrieval of images and sentences through a deep, multi-modal embedding of visual and natural language data. Unlike previous models that directly map images or sentences into a common embedding space, our model works on a finer level and embeds fragments of images (objects) and fragments of sentences (typed dependency tree relations) into a common space. We then introduce a structured max-margin objective that allows our model to explicitly associate these fragments across modalities. Extensive experimental evaluation shows that reasoning on both the global level of images and sentences and the finer level of their respective fragments improves performance on image-sentence retrieval tasks. Additionally, our model provides interpretable predictions for the image-sentence retrieval task since the inferred inter-modal alignment of fragments is explicit.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, background): “Over time work explored many combinations of training objective, transfer, and more expressive models and steadily improved performance (Frome et al., 2013; Socher et al., 2014; Karpathy et al., 2014; Kiros et al., 2014; Faghri et al., 2017).”
End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, neutral): “We compare the NCE based approach (ours) to the frequently used max margin ranking loss [19, 21, 33, 40, 51, 55, 79, 80, 83, 84] and a binary classification loss (i.”
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, methodology): “We train our embedding model using the maxmargin ranking loss [21, 32, 54, 55, 64].”
Webly Supervised Joint Embedding for Cross-Modal Image-Text Retrieval (2018, methodology): “Many prior approaches have utilized pairwise ranking loss as the objective for learning joint embedding between visual input and textual input [24, 30, 55, 60].”
Learning Visual N-Grams from Web Data (2016, methodology): “Like other language models, our models perform worse on the Flickr-30K dataset than dedicated retrieval models [30, 33, 60, 65].”
Learning Visual Features from Large Weakly Supervised Data (2015, background): “Some works co-embed images and words [14], whereas others co-embed images and sentences or n-grams [13, 23, 53].”
Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models (2014, methodology): “The deep visual semantic embedding model [5] was proposed as a way of performing zero-shot object recognition and was used as a baseline by [15].”
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision (2021, methodology): “…et al., 2013; Faghri et al., 2018) and improved versions (e.g., leveraging object detectors, dense feature maps, or multi-attention layers) (Socher et al., 2014; Karpathy et al., 2014; Kiros et al.; Nam et al., 2017; Li et al., 2019; Messina et al., 2020; Chen et al., 2020a) have been proposed.”
Microsoft COCO Captions: Data Collection and Evaluation Server (2015, background): “The automatic generation of captions for images is a long standing and challenging problem in artiﬁcial intelligence [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19].”
Long-term recurrent convolutional networks for visual recognition and description (2014, methodology): “We first show the effectiveness of our model by quantitatively evaluating it on the image and caption retrieval tasks proposed by [26] and seen in [25, 15, 36, 8, 18].”
Show and tell: A neural image caption generator (2014, background): “Such multiplicative gates make it possible to train the LSTM robustly as these gates deal well with exploding and vanishing gradients [10].”
Deep Visual-Semantic Alignments for Generating Image Descriptions (2014, neutral): “We then described a multimodal Recurrent Neural Network architecture that generates textual descriptions based on image regions, and evaluated its performance with fullframe and region-level experiments.”
CIDEr: Consensus-based image description evaluation (2014, methodology): “Recent works have utilized techniques in deep learning to learn joint embeddings of text and image fragments [20].”
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models (2015, methodology): “n seen individually at training time, but not in that combination (Devlin et al. 2015). Some recent works do try to ﬁnd correspondences between image regions and parts of sentences (Fang et al. 2015; Karpathy et al. 2014; Karpathy and Fei-Fei 2015; Xu et al. 2015), but they treat such correspondence”
Neural Module Networks (2015, background): “This grounding task has previously been approached in [18, 32, 17, 20, 14], where the authors tried to localize phrases in an image.”
Learning Deep Structure-Preserving Image-Text Embeddings (2015, methodology): “As a more powerful objective function, a few other works have proposed a bi-directional ranking loss that, in addition to ensuring that correct sentences for each training image get ranked above incorrect ones, also ensures that for each sentence, the image described by that sentence gets ranked abo”
Visual7W: Grounded Question Answering in Images (2015, background): “Pioneer work in image captioning [4, 5, 13, 45, 47], sentence-based image retrieval [14, 40] and visual QA [1, 7, 36] shows promising results.”
From captions to visual concepts and back (2014, background): “A similar fine-grained analysis of sentences and images is done for retrieval in the context of neural networks [17].”
Stacked Cross Attention for Image-Text Matching (2018, background): “We empirically find it beneficial to threshold the similarities at zero [20] and normal-”
Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN) (2014, methodology): “For the vision part, we use the pre-trained AlexNet (Krizhevsky et al. (2012)) or the VggNet (Simonyan & Zisserman (2014)) on ImageNet dataset (Russakovsky et al.”