Multi-Similarity Loss With General Pair Weighting for Deep Metric Learning

By Xun Wang, Xintong Han, Weilin Huang, Dengke Dong, Matthew R. Scott. 2019

909 citations recorded by the source corpus.

A family of loss functions built on pair-based computation have been proposed in the literature which provide a myriad of solutions for deep metric learning. In this pa-per, we provide a general weighting framework for under-standing recent pair-based loss functions. Our contributions are three-fold: (1) we establish a General Pair Weighting (GPW) framework, which casts the sampling problem of deep metric learning into a unified view of pair weighting through gradient analysis, providing a powerful tool for understanding recent pair-based loss functions; (2) we show that with GPW, various existing pair-based methods can be compared and discussed comprehensively, with clear differences and key limitations identified; (3) we propose a new loss called multi-similarity loss (MS loss) under the GPW,which is implemented in two iterative steps (i.e., mining and weighting). This allows it to fully consider three similarities for pair weighting, providing a more principled approach for collecting and weighting informative pairs. Finally, the proposed MS loss obtains new state-of-the-art performance on four image retrieval benchmarks, where it outperforms the most recent approaches, such as ABE[14] and HTL[4], by a large margin, e.g.,60.6%→65.7%on CUB200,and 80.9%→88.0%on In-Shop Clothes Retrieval datasetat Recall@1.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Continual Learning through Retrieval and Imagination (2022, methodology): “Here, we employ a pair-based metric learning loss, Multi-Similarity (MS) loss (Wang et al. 2019a), as our loss function ℓ : where S i,k denotes the similarity of a pair { x i , x k } ; P i and N i are the positive and negative sets for the anchor x i ; N b is the number of training samples in a…”
Semantic Drift Compensation for Class-Incremental Learning (2020, methodology): “Additionally, we investigated two newer versions of metric losses: Multi-similarity [43] and Angular loss [42] on CUB-200-2011 with a class-IL setting.”
ProxyNCA++: Revisiting and Revitalizing Proxy Neighborhood Component Analysis (2020, methodology): “[32] categorize this group as paired-based DML.”
Deep Metric Learning: A Survey (2019, background): “Unlike all other loss metrics in the literature, the authors in [116] benefited from multi-similarity loss while using self-similarity, negative relative similarity, and positive relative similarity under the general pair weighting framework to capture the similarity between samples.”
Asymmetric metric learning for knowledge transfer (2020, methodology): “Originating in metric learning, loss functions based on pairwise distances or similarities [18, 71, 46, 72, 5] are paramount in representation learning.”
A Spatial-Temporal Attention-Based Method and a New Dataset for Remote Sensing Image Change Detection (2020, background): “be closer, while dissimilar ones are pushed apart from each other [59].”
ABO: Dataset and Benchmarks for Real-World 3D Object Understanding (2021, methodology): “…on our multi-view cross-domain retrieval benchmark, we use PyTorch Metric Learning [2] implementations that cover the main approaches to DML: NormSoft-max [70] (classification-based), ProxyNCA [50] (proxy-based) and Contrastive, TripletMargin, NTXent [11] and Multi-similarity [63] (tuple-based).”
Characterizing Generalization under Out-Of-Distribution Shifts in Deep Metric Learning (2021, methodology): “To analyze the zero-shot transfer capabilities of DML models, we consider representative approaches making use of the following concepts: (i) surrogate ranking tasks and tuple mining heuristics (Margin loss with distance-based sampling [71] and Multisimilarity loss [70]), (ii) geometric constraints ”
Self-Alignment Pretraining for Biomedical Entity Representations (2020, background): “…a plethora of text mining systems in the medical domain, facilitating applications such as literature search (Lee et al., 2016), clinical decision making (Roberts et al., 2015) and relational knowledge discovery (e.g. chemical-disease, drug-drug and protein-protein relations) (Wang et al., 2018).”
Training Vision Transformers for Image Retrieval (2021, neutral): “Other pair-based losses rely on the soft-max function (Goldberger et al., 2005; Sohn, 2016; Wu et al., 2018; Wang et al., 2019), allowing for more comparisons between different positive and negative pairs.”
A Metric Learning Reality Check (2020, background): “Softmax) [58,31,72] 2017 Classification CosFace [57,59] 2018 Classification ArcFace [11] 2019 Classification FastAP [3] 2019 Embedding Signal to Noise Ratio Contrastive (SNR) [70] 2019 Embedding MultiSimilarity (MS) [62] 2019 Embedding MS+Miner [62] 2019 Embedding SoftTriple [41] 2019 Classification”
Proxy Anchor Loss for Deep Metric Learning (2020, methodology): “7 - - MS [34] 74.”
Prime-Aware Adaptive Distillation (2020, methodology): “For the teacher, we adopt the GoogLeNet [31] trained by multi-similarity loss [35], which provides a higher R@1 of 64.”
Revisiting Training Strategies and Generalization Performance in Deep Metric Learning (2020, background): “On average, margin (Wu et al., 2017) and multisimilarity loss (Wang et al., 2019a) offer the best performance across datasets.”
Channel Augmented Joint Learning for Visible-Infrared Recognition (2021, methodology): “Compared to the widely-used online hard triplet mining [15] and multi-similarity loss [35], our strategy generally performs better under various settings.”
Circle Loss: A Unified Perspective of Pair Similarity Optimization (2020, methodology): “On these datasets, majority methods [19, 18, 3, 20, 13, 34] adopt the encouraged setting of learning with pair-wise labels.”