VSE++: Improving Visual-Semantic Embeddings with Hard Negatives

By Fartash Faghri, David J. Fleet, Jamie Ryan Kiros, Sanja Fidler. 2017

1,393 citations recorded by the source corpus.

Subfields: Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Multimodal Learning

1. Zhe Gan
2. Lijuan Wang
3. Jianfeng Gao
4. Linjie Li
5. Zicheng Liu
6. Mohit Bansal
7. Jingjing Liu
8. Jianwei Yang

In Representation & Self-Supervised Learning

1. Jianfeng Gao
2. Ishan Misra
3. Yoshua Bengio
4. Andrew Zisserman
5. Yann LeCun
6. Andrea Vedaldi
7. Armand Joulin
8. Quoc V. Le

Citations in the corpus

• Learning Transferable Visual Models From Natural Language Supervision (2021, background): “Over time work explored many combinations of training objective, transfer, and more expressive models and steadily improved performance (Frome et al., 2013; Socher et al., 2014; Karpathy et al., 2014; Kiros et al., 2014; Faghri et al., 2017).”
• Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training (2019, methodology): “We follow the data split in (Faghri et al. 2017) to add 30,504 images that were originally in the validation set of MSCOCO.”
• Combined Scaling for Zero-shot Transfer Learning (2021, methodology): “Similar to CLIP and ALIGN, our work also learns the mapping between images and texts, which is related to many previous works, such as (Hironobu et al., 1999; Weston et al., 2010; Socher and Fei-Fei, 2010; Socher et al., 2013; Hodosh et al., 2013; Frome et al., 2013; Norouzi et al., 2013; Kiros et a”
• LiT: Zero-Shot Transfer with Locked-image text Tuning (2021, background): “Another approach, which we adopt in this work, is to learn an alignment between image and text embedding spaces [6, 15, 21, 22, 31, 70].”
• Align before Fuse: Vision and Language Representation Learning with Momentum Distillation (2021, background): “The second category focuses on learning separate unimodal encoders for image and text [22, 23, 6, 7].”
• Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision (2021, methodology): “Similar objectives has been applied to learning visual-semantic embeddings (VSE) (Frome et al., 2013; Faghri et al., 2018).”
• Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers (2021, methodology): “…we can achieve the best zero-shot retrieval results us-ing a classification loss which uses only one negative example per image–text pair (compare to a contrastive loss with 16384 negatives used in Tian et al., 2019) and also removes the need for mining more hard negatives (Faghri et al., 2017).”
• ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision (2021, methodology): “The visual semantic embedding (VSE) models such as VSE++ (Faghri et al., 2017) and SCAN (Lee et al., 2018) belong to Figure 2a.”
• Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers (2020, background): “…and language tasks, including Visual Question Answering (VQA) and Natural Language for Visual Rea-10 Model TR IR R@1 R@5 R@10 R@1 R@5 R@10 VSE++[10] 52.9 80.5 87.2 39.6 70.1 79.5 SCAN[17] 67.4 Natural Language for Visual Reasoning for Real Natural Language for Visual Reasoning for Real (NLVR 2…”
• Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning (2021, methodology): “Model Backbone TR IR R@1 R@5 R@10 R@1 R@5 R@10 VSE++[12] R152 52.9 80.5 87.2 and text-to-image retrieval ( IR ), and are conducted on Flickr30K [45] and MSCOCO [25] datasets.”
• Support-set bottlenecks for video-text representation learning (2020, methodology): “0 − − − − VSE++ (Faghri et al., 2018) Contrastive Loss.”
• Semantics Disentangling for Text-To-Image Generation (2019, methodology): “Inspired by [39, 8, 7, 37], the visual-semantic embedding (VSE) module is adopted for mutual fusion of word features and visual features, as shown in Fig.”