Smart Mining for Deep Metric Learning

By Ben Harwood, B. V. Kumar, Gustavo Carneiro, Ian David Reid, Tom Drummond. 2017

373 citations recorded by the source corpus.

To solve deep metric learning problems and producing feature embeddings, current methodologies will commonly use a triplet model to minimise the relative distance between samples from the same class and maximise the relative distance between samples from different classes. Though successful, the training convergence of this triplet model can be compromised by the fact that the vast majority of the training samples will produce gradients with magnitudes that are close to zero. This issue has motivated the development of methods that explore the global structure of the embedding and other methods that explore hard negative/positive mining. The effectiveness of such mining methods is often associated with intractable computational requirements. In this paper, we propose a novel deep metric learning method that combines the triplet model and the global structure of the embedding space. We rely on a smart mining procedure that produces effective training samples for a low computational cost. In addition, we propose an adaptive controller that automatically adjusts the smart mining hyper-parameters and speeds up the convergence of the training process. We show empirically that our proposed method allows for fast and more accurate training of triplet ConvNets than other competing mining methods. Additionally, we show that our method achieves new state-of-the-art embedding results for CUB-200-2011 and Cars196 datasets.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, methodology): “These methods need careful treatment of negative pairs [13] by either relying on large batch sizes [8, 12], memory banks [9] or customized mining strategies [14, 15] to retrieve the negative pairs.”
Unsupervised Embedding Learning via Invariant and Spreading Instance Feature (2019, methodology): “Following existing works on supervised deep embedding learning [13, 32], the retrieval performance and clustering quality of the testing set are evaluated.”
Continual Prototype Evolution: Learning Online from Non-Stationary Data Streams (2020, background): “Further, in deep embedding learning most commonly pairs [15] and triplets [17] of samples are considered in contrastive”
Meta-Transfer Learning for Few-Shot Learning (2018, background): “Training on more confusing data enables the model to achieve higher robustness and better performance [4, 15, 7].”
Imbalance Problems in Object Detection: A Review (2019, background): “[168], the hardness of the examples in the latter epochs is increased.”
Heated-Up Softmax Embedding (2018, methodology): “ss (Song et al., 2016), [3] Learnable Structured Clustering (Song et al., 2017); [4] Deep Clustering Learning without spectral learning (Law et al., 2017); [5] Deep Metric Learning with Smart Mining (Harwood et al., 2017) and [6] ProxyNCA (Movshovitz-Attias et al., 2017). For the Table 5: R@1(%) for”
When and why vision-language models behave like bags-of-words, and what to do about it? (2022, background): “Negative mining and contrastive learning : Using hard negatives has been successful in improving representation learning (Harwood et al., 2017; Wu et al., 2017; Ge, 2018).”
Revisiting Training Strategies and Generalization Performance in Deep Metric Learning (2020, background): “While previous works only studied this process in the specific context of triplet mining strategies for ranking-based objectives (Wu et al., 2017; Harwood et al., 2017), we examine the model-agnostic case of sampling informative mini-batches.”
Hard negative examples are hard, but useful (2020, background): “in fact, hard negative triplets are essentially errors in the trained image mappings), and considering challenging combinations of images has proven critical in triplet based distance metric learning [4,6,8,16,21]. But challenges in optimizing with the hardest negative examples are widely reported i”
Characterizing Generalization under Out-Of-Distribution Shifts in Deep Metric Learning (2021, methodology): “However, due to the exponentially increased complexity of tuple sampling spaces, these methods are usually also combined with tuple sampling objectives, relying on predefined or learned heuristics to avoid training over tuples that are too easy or too hard [57, 72] or reducing tuple redudancy encoun”
Asymmetric metric learning for knowledge transfer (2020, background): “The large number of potential tuples gives rise to mining [20, 75] and memory [73, 76] mechanisms.”
Hard Negative Mixing for Contrastive Learning (2020, background): “Hard negatives are critical for contrastive learning [1, 19, 23, 30, 37, 44, 48].”
Proxy Anchor Loss for Deep Metric Learning (2020, background): “Pair Contrastive [2, 4, 9] O(M) Triplet (Semi-Hard) [25] O(M/B) Triplet (Smart) [10] O(M) N -pair [27] O(M) Lifted Structure [29] O(M)”
Mining on Manifolds: Metric Learning Without Labels (2018, methodology): “Our goal is to mine such training pairs without any supervision [9, 62, 18, 18, 34], without any complementary computer vision system [15, 42] or assumptions on the nature of the training data [64, 65, 11].”
A Metric Learning Reality Check (2020, background): “onstruction, so that each batch is made to contain the most informative samples. This might be accomplished by storing lists of hard negatives [49], doing a nearest neighbors search before each epoch [17], or before each iteration [51]. In contrast, online mining nds hard pairs or triplets within ea”
Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning (2021, background): “Meanwhile, since hard pairs are more informative for contrastive learning [20], Core-tuning further assigns higher importance weights to hard positive pairs based on a new focal contrastive loss.”