Learning Deep Structure-Preserving Image-Text Embeddings

By Liwei Wang, Yin Li, Svetlana Lazebnik. 2015

838 citations recorded by the source corpus.

This paper proposes a method for learning joint embeddings of images and text using a two-branch neural network with multiple layers of linear projections followed by nonlinearities. The network is trained using a large-margin objective that combines cross-view ranking constraints with within-view neighborhood structure preservation constraints inspired by metric learning literature. Extensive experiments show that our approach gains significant improvements in accuracy for image-to-text and text-to-image retrieval. Our method achieves new state-of-the-art results on the Flickr30K and MSCOCO image-sentence datasets and shows promise on the new task of phrase localization on the Flickr30K Entities dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

UNITER: Learning UNiversal Image-TExt Representations (2019, neutral): “ of an Image Embedder, a Text Embedder and a multi-layer Transformer, learned through four pre-training tasks studied learning latent alignment between words and image regions for ImageText Retrieval [47] and Referring Expression Comprehension [15]. While each of these models has pushed the state of”
Learning Visual Representations with Caption Annotations (2020, background): “Vision and language (VL) have been jointly leveraged to learn cross-modal representations for various VL tasks, such as crossmodal retrieval [21,61], visual question answering [25], captioning [56] or visual grounding [11,30].”
Self-Supervised MultiModal Versatile Networks (2020, background): “With larger datasets [48, 69, 71, 92, 96], many works have profited from learning such a joint visual-textual space [17, 19, 29, 30, 41, 53, 57, 63, 68, 84, 85, 87, 88, 93].”
End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, methodology): “We compare the NCE based approach (ours) to the frequently used max margin ranking loss [19, 21, 33, 40, 51, 55, 79, 80, 83, 84] and a binary classification loss (i.”
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, methodology): “Examples of tasks that still represent a significant challenge for current artificial systems include text-to-video retrieval [25, 32, 54, 55, 63], text-based action or event localization [15], video captioning [36, 61], and video question answering [51, 63].”
Webly Supervised Joint Embedding for Cross-Modal Image-Text Retrieval (2018, methodology): “, cross-modal retrieval [18, 31, 38, 55], image captioning [23, 36], image classication [10, 12, 20] video summarization [4, 41].”
Learning Visual N-Grams from Web Data (2016, methodology): “Like other language models, our models perform worse on the Flickr-30K dataset than dedicated retrieval models [30, 33, 60, 65].”
Towards Language-Free Training for Text-to-Image Generation (2021, background): “Unlike the aforementioned works, another line of works focus on the way of learning visual representation from natural language supervisions, including both generative [5] and discriminative [48, 49, 58] methods.”
Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval (2021, background): “Joint visual-text models have become increasingly popular as they enable a wide suite of downstream tasks, including text-to-visual retrieval [36, 38, 43, 66], visual captioning [29, 65, 73], and visual question and answering [4, 33].”
Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers (2021, methodology): “Despite lacking a multimodal attention mechanism, this approach has been popular in image and video domains because of its simplicity and effectiveness for retrieval applications (e.g., Gong et al., 2014; Wang et al., 2016; Chowdhury et al., 2018; Miech et al., 2018).”
Unified Contrastive Learning in Image-Text-Label Space (2022, background): “(ii) Another line of works focuses on learning transferable visual representation from natural language supervisions, including generative [12, 48] and contrastive methods [16,29,47,59,60,74].”
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models (2015, methodology): “Phrase Correspondences for Richer Image-to-Sentence Models 13 Methods R@1 R@5 R@10 (a) Zhang et al. (2016) 27.0 49.9 57.7 Hu et al. (2016) - VGG19 27.8 – 62.9 Rohrbach et al. (2016) - VGG19 41.56 – – Wang et al. (2016b) - Fast RCNN 42.08 – – Wang et al. (2016a) - Fast RCNN 43.89 64.46 68.66 Rohrbach”
Bilinear Attention Networks (2018, neutral): “…[11] Edge Boxes [42] 27.8 - 62.9 76.9 Rohrbach et al. [26] Fast RCNN [7] 42.43 - - 77.90 Wang et al. [33] Fast RCNN [7] 42.08 - - 76.91 Wang et al. [32] Fast RCNN [7] 43.89 64.46 68.66 76.91 Rohrbach et al. [26] Fast RCNN [7] 48.38 - - 77.90 Fukui et al. [6] Fast RCNN [7] 48.69 - - - Plummer et…”
Stacked Cross Attention for Image-Text Matching (2018, background): “A rich line of studies have explored mapping whole images and full sentences to a common semantic vector space for image-text matching [22, 38, 2, 39, 23, 27, 44, 10, 33, 13, 9, 8, 11].”
TVQA: Localized, Compositional Video Question Answering (2018, background): “For image-based object grounding, there has been much work on phrase grounding (Plummer et al., 2015; Wang et al., 2016b; Rohrbach et al., 2016) and referring expression comprehension (Hu et al., 2016; Yu et al., 2016; Nagaraja et al., 2016; Yu et al., 2017, 2018b).”