VSE++: Improved Visual-Semantic Embeddings

By Fartash Faghri, David J. Fleet, Jamie Ryan Kiros, Sanja Fidler. 2017

189 citations recorded by the source corpus.

This paper investigates the problem of image-caption retrieval using joint visual-semantic embeddings. We introduce a very simple change to the loss function used in the original formulation by Kiros et al. (2014), which leads to drastic improvements in the retrieval performance. In particular, the original paper uses the rank loss which computes the sum of violations across the negative training examples. Instead, we penalize the model according to the hardest negative examples. We then make several additional modifications according to the current best practices in image-caption retrieval. We showcase our model on the MS-COCO and Flickr30K datasets through comparisons and ablation studies. On MS-COCO, we improve caption retrieval by 21% in R@1 with respect to the original formulation. Our results outperform the state-of-the-art results by 8.8% in caption retrieval and 11.3% in image retrieval at R@1. On Flickr30K, we more than double R@1 as reported by Kiros et al. (2014) in both image and caption retrieval, and achieve near state-of-the-art performance. We further show that similar improvements also apply to the Order-embeddings by Vendrov et al. (2015) which builds on a similar loss function.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks (2020, neutral): “8 - - - - - VSE++ [7] - 64.”
Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training (2019, methodology): “We follow the data split in (Faghri et al. 2017) to add 30,504 images that were originally in the validation set of MSCOCO.”
Webly Supervised Joint Embedding for Cross-Modal Image-Text Retrieval (2018, methodology): “In this work, we use cosine similarity in the representation space to calculate similarity, which is widely used in learning image-text embedding and shown to be very effective in many prior works [8, 30, 60].”
Stacked Cross Attention for Image-Text Matching (2018, methodology): “Following the split in [19, 10], we use 1,000 images for validation and 1,000 images for testing and the rest for training.”
Learning Joint Embedding with Multimodal Cues for Cross-Modal Video-Text Retrieval (2018, methodology): “We also implement and compare results with state-of-the-art image-embedding approachVSE[18] andVSEPP[7] using ResNet152 feature as the video feature.”
Zero-Shot Object Detection (2018, methodology): “These methods can be used for cross-modal retrieval tasks [9], or for transferring classiﬁers between modalities.”
Use What You Have: Video retrieval using representations from collaborative experts (2019, background): “[36] observe that a simple adaptation of a state-of-the-art image-text embedding method [17] by mean-pooling features from video frames provides a better result than many prior video-text retrieval approaches [12, 39].”
Dual Encoding for Zero-Example Video Retrieval (2018, methodology): “While focusing on video retrieval, we provide an additional experiment on Flickr30k [35] and MS-COCO [19] to investigate if the VSE++ model [8], the state-of-the-art for image-text retrieval, can be improved by replacing its GRU based encoding by the proposed encoding at the text side.”
Learning the Best Pooling Strategy for Visual Semantic Embedding (2020, methodology): “VSE++ is the fundamental VSE method as described in § 4, we re-implement it (denoted as Our: VSE++) and apply it on latest feature extractors (e.g., BUTD image features, BERT, etc.).”
Talking Face Generation by Adversarially Disentangled Audio-Visual Representation (2018, methodology): “Here we report the R @1 , R @10 and Med R measurements which is the same as [14].”
Dual-Path Convolutional Image-Text Embedding (2017, result): “Compared with a recent arXiv work, VSE++ [3], our result is also competitive.”
Look, Imagine and Match: Improving Textual-Visual Cross-Modal Retrieval with Generative Models (2017, background): “Image-to-Text Text-to-Image Model R@1 R@10 Med r R@1 R@10 Med r *VSE++ [4] 52.”
More Grounded Image Captioning by Distilling Image-Text Matching Model (2020, methodology): “The key motivation of our knowledge distillation is that compared to the caption generation task, the image-text matching task [9, 24] is a more wellposed one, because 1) the latter doesn’t have to take the sentence grammar and fluency into account, and 2) the training loss for the latter’s metric (”
Adversarial Attacks on Deep Learning Models in Natural Language Processing: A Survey (2019, unknown): “Visual-Semantic Embeddings [125] N black-box VSE++ [33] –”
Learning Visually Grounded Sentence Representations (2017, methodology): “Note that while this work underwent review, better methods have been published, most notably VSE++ (Faghri et al., 2017).”
CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrieval (2019, methodology): “VSE++ [5] jointly embeds image features and sentence features into the same embedding space and calculates image-sentence similarities as distances of embedded features, and train the model with ranking loss with hardest negative samples in a mini-batch.”
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis (2020, methodology): “3) Evaluation: We compare KT-GAN and other state-ofthe-art algorithms using four measures: (1) Inception Score (IS) [27]; (2) Fréchet Inception Distance (FID) [12]; (3) We also compare the Rank-1 score in text-to-image retrieval [5]; (4) Human perceptual test.”
Engaging Image Chat: Modeling Personality in Grounded Dialogue (2018, methodology): “Again, a variety of sequence-to-sequence (Vinyals et al., 2015; Xu et al., 2015; Anderson et al., 2018) and retrieval models (Gu et al., 2017; Faghri et al., 2017; Nam et al., 2016) have been applied.”
Semantics-Enhanced Adversarial Nets for Text-to-Image Synthesis (2019, methodology): “We use the trained model in [5] to evaluate the semantic consistency, and select the Rank-1 as our evaluation score.”