Sampling Matters in Deep Embedding Learning

By Raghavan Manmatha, Chaoxia Wu, Alex Smola, Philipp Krahenbuhl. 2017

1,013 citations recorded by the source corpus.

Deep embeddings answer one simple question: How similar are two images? Learning these embeddings is the bedrock of verification, zero-shot learning, and visual search. The most prominent approaches optimize a deep convolutional network with a suitable loss function, such as contrastive loss or triplet loss. While a rich line of work focuses solely on the loss functions, we show in this paper that selecting training examples plays an equally important role. We propose distance weighted sampling, which selects more informative and stable examples than traditional approaches. In addition, we show that a simple margin based loss is sufficient to outperform all other loss functions. We evaluate our approach on the Stanford Online Products, CAR196, and the CUB200-2011 datasets for image retrieval and clustering, and on the LFW dataset for face verification. Our method achieves state-of-the-art performance on all of them.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Self-Supervised MultiModal Versatile Networks (2020, methodology): “e search across modalities [36]. This idea is at the core of our versatile networks. With larger datasets [47, 69, 71, 92, 96], many works have proﬁted from learning such a joint visual-textual space [16, 18, 28, 29, 40, 52, 56, 62, 68, 84, 85, 87, 88, 93]. Recently, instructional videos became a po”
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, methodology): “These methods need careful treatment of negative pairs [13] by either relying on large batch sizes [8, 12], memory banks [9] or customized mining strategies [14, 15] to retrieve the negative pairs.”
End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, methodology): “We compare the NCE based approach (ours) to the frequently used max margin ranking loss [19, 21, 33, 40, 51, 55, 79, 80, 83, 84] and a binary classification loss (i.”
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, background): “A common approach to model vision and language is learning a joint embedding space where visual and textual cues are adjacent if and only if they are semantically similar [6, 8, 10, 11, 25, 32, 35, 37, 38, 59, 54, 55, 57].”
Webly Supervised Joint Embedding for Cross-Modal Image-Text Retrieval (2018, background): “Recently, focusing on hard-negatives has been shown to be effective in learning joint embeddings [8, 44, 56, 60].”
Unsupervised Embedding Learning via Invariant and Spreading Instance Feature (2019, background): “Most of them are designed on top of pairwise [12, 30] or triplet relationships [13, 29].”
Learning a Text-Video Embedding from Incomplete and Heterogeneous Data (2018, background): “One way to leverage language in vision is to find a joint embedding space for both visual and textual modalities [7–9, 19–23].”
Relational Knowledge Distillation (2019, methodology): “2 and use distance-weighted sampling [42] for triplets.”
MultiGrain: a unified image embedding for classes and instances (2019, methodology): “[45] proposed an effective method that addresses these difficulties.”
Learning Joint Embedding with Multimodal Cues for Cross-Modal Video-Text Retrieval (2018, background): “Moreover, selecting the hardest negatives in practice may often lead to a collapsed model and semi-hard negative mining helps to mitigate this issue [21, 28].”
Recurrent Pixel Embedding for Instance Grouping (2017, background): “[87] argue that the constant slope of the margin loss is more robust, e.”
Not All Samples Are Created Equal: Deep Learning with Importance Sampling (2018, methodology): “On the other hand, for deep neural networks, sample selection methods were mainly employed to generate hard negative samples for embedding learning problems or to tackle the class imbalance problem (Schroff et al., 2015; Wu et al., 2017; Simo-Serra et al., 2015).”
Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformers (2021, methodology): “A large number of vision and language retrieval models [15, 19, 20, 36, 50, 51, 55, 59, 71, 72, 74, 75, 77] use a dual encoder architecture where the text and vision inputs are separately embedded into a joint space.”
Training Vision Transformers for Image Retrieval (2021, neutral): “As pointed out by Wu et al. (2017), very hard negatives can lead to noisy gradients.”
Cluster Contrast for Unsupervised Person Re-Identification (2021, background): “To solve this problem, distance weighted sampling [43] selects more informative and stable examples for training and batch hard triplet loss [17] is presented to mine the hardest triplet in each batch.”
Mancs: A Multi-task Attentional Network with Curriculum Sampling for Person Re-Identification (2018, background): “However, it is easy to collapse according to [26].”
Hard Negative Mixing for Contrastive Learning (2020, background): “Hard negatives are critical for contrastive learning [1, 19, 23, 30, 37, 44, 48].”
Proxy Anchor Loss for Deep Metric Learning (2020, background): “To resolve these issues, learning with the pair-based losses often entails tuple sampling techniques [10, 25, 37, 40], which however have to be tuned by hand and may increase the risk of overfitting.”
Heterogeneous Attention Network for Effective and Efficient Cross-modal Retrieval (2021, background): “Nevertheless, the max triplet loss only considers the hardest negative pairs and thus it might suffer from mode collapse [43].”