Improving Image-Sentence Embeddings Using Large Weakly Annotated Photo Collections

By Yunchao Gong, Liwei Wang, Micah Hodosh, Julia Hockenmaier, Svetlana Lazebnik. 2014

315 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Learning Video Representations from Textual Web Supervision (2020, methodology): “The most commonly-used approach is to use image search results, and to label each image with the query that was used to ﬁnd it [65, 15, 56, 6, 11, 13, 21, 10, 18, 35].”
Self-Supervised MultiModal Versatile Networks (2020, background): “With larger datasets [48, 69, 71, 92, 96], many works have profited from learning such a joint visual-textual space [17, 19, 29, 30, 41, 53, 57, 63, 68, 84, 85, 87, 88, 93].”
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, background): “A common approach to model vision and language is learning a joint embedding space where visual and textual cues are adjacent if and only if they are semantically similar [6, 8, 10, 11, 25, 32, 35, 37, 38, 59, 54, 55, 57].”
Webly Supervised Joint Embedding for Cross-Modal Image-Text Retrieval (2018, methodology): “Most relevant to our work is [13], where authors constructed a dictionary by taking a few thousand most common words and represent text as tf-idf weighted bag of words (BoW) vectors that ignore word order and represents each caption as a vector of word frequencies.”
Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models (2014, background): “Several bi-directional approaches to ranking images and captions have also been proposed, based off of kernel CCA [3], normalized CCA [19] and dependency tree recursive networks [6].”
Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics (2013, methodology): “In [Gong et al., 2014], we present the first image description evaluation on Flickr30K, and show that a CCA-based approach can benefit significantly from the incorporation of Flickr meta-data.”
Microsoft COCO Captions: Data Collection and Evaluation Server (2015, background): “The automatic generation of captions for images is a long standing and challenging problem in artiﬁcial intelligence [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19].”
Show and tell: A neural image caption generator (2014, background): “The definition of the gates and cell update and output are as follows: it = σ(Wixxt +Wimmt−1) (4) ft = σ(Wfxxt +Wfmmt−1) (5) ot = σ(Woxxt +Wommt−1) (6) ct = ft ct−1 + it h(Wcxxt +Wcmmt−1)(7) mt = ot ct (8) pt+1 = Softmax(mt) (9) where represents the product with a gate value, and the various W…”
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models (2015, methodology): “We build on the Flickr30k dataset [40], a popular benchmark for caption generation and retrieval [1, 4, 6, 10, 17, 16, 19, 20, 23, 25, 37, 38].”
Learning Deep Structure-Preserving Image-Text Embeddings (2015, methodology): “As mentioned earlier, in our past experience we have found CCA to be surprisingly hard to beat with more complex methods [15, 37].”
Learning a Recurrent Visual Representation for Image Caption Generation (2014, background): “Whereas learning the mapping between image patches and single text labels remains a popular topic in computer vision [18, 8, 9], there is a growing interest in using entire sentence descriptions together with pixels to learn joint embeddings [13, 32, 16, 10].”
Grounding of Textual Phrases in Images by Reconstruction (2015, methodology): “[1] report the localization performance of their proposed CCA embedding [14] approach.”
Learning to Answer Questions from Image Using Convolutional Neural Network (2015, background): “Recently, there is an increasing research area for the multimodal learning between image and language (Kiela et al., 2014; Gong et al., 2014; Srivastava et al., 2012a; Srivastava et al., 2012b).”
Image Captioning with Semantic Attention (2016, background): “It has been shown that these weakly annotated images can be exploited to learn visual concepts [38], text-image embedding [14] and image captions [7].”
Where to Look: Focus Regions for Visual Question Answering (2015, background): “Many recent works in tying text to images have explored the task of automated image captioning [10, 7, 22, 12, 13, 14, 6, 4, 21].”
Show and Tell: Lessons Learned from the 2015 MSCOCO Image Captioning Challenge (2016, methodology): “An “encoder” RNN reads the source sentence and transforms it into a rich ﬁxed-length vector representation, which in turn in used as the initial hidden state of a “decoder” RNN that generates the target sentence.”
Multimodal Convolutional Neural Networks for Matching Image and Sentence (2015, background): “Describing the image with natural sentences is useful for image annotation and captioning [8, 21, 26], while retrieving image with a natural language as query is more natural for image search [13, 16].”