A Metric Learning Reality Check

By Kevin Musgrave, Serge J. Belongie, Ser-Nam Lim. 2020

537 citations recorded by the source corpus.

Deep metric learning papers from the past four years have consistently claimed great advances in accuracy, often more than doubling the performance of decade-old methods. In this paper, we take a closer look at the field to see if this is actually true. We find flaws in the experimental methodology of numerous metric learning papers, and show that the actual improvements over time have been marginal at best.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Are we done with ImageNet? (2020, background): “Several studies have revisited common computer vision benchmarks [16, 17, 18, 19] and, regarding ImageNet [1] speciﬁcally, identiﬁed various sources of bias and noise [20, 21, 22].”
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision (2021, background): “This is one of the most effective loss functions for both self-supervised (Chen et al., 2020b) and supervised (Zhai & Wu, 2019; Musgrave et al., 2020) representation learning.”
On the Gap between Adoption and Understanding in NLP (2021, background): “Methodological errors can slow research; as shown by Musgrave et al. (2020), due to methodological ﬂaws in the experiments, the increase of performance in metric learning was wrongly re-ported in several papers.”
The Benchmark Lottery (2021, background): “[Machado et al., 2018, Bouthillier et al., 2021, Balduzzi et al., 2018, Bouthillier et al., 2019, Musgrave et al., 2020], or “What are the best practices for fair benchmarking?””
VIGOR: Cross-View Image Geo-localization beyond One-to-one Retrieval (2020, background): “Metric learning literature [15, 20, 13] has revealed the importance of mining hard samples during training, as the model would suffer from poor convergence when most samples barely contribute to the total loss.”
Project CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks (2021, methodology): “We refer the reader to [37] for details on how MAP@R is computed.”
SphereFace Revived: Unifying Hyperspherical Face Recognition (2021, background): “The development of deep FR is also closely related to deep metric learning [17], [18], [19], [20], [21], [22], [23], [24], [25], [26], [27], [28].”
DeepEMD: Differentiable Earth Mover's Distance for Few-Shot Learning (2020, background): “hensive empirical study in [124], where existing deep metric”
DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems (2020, methodology): “In light of recent concerns about poor reproducibility of published results [8, 33, 38], we conducted a fair and comprehensive experimental study with extensive hyper-parameter search to properly tune all the baselines and proposed approaches.”
What Can We Do to Improve Peer Review in NLP? (2020, background): “Inter alia: unfair comparisons (Musgrave et al., 2020), dependence on non-architecture-related factors (Dodge et al.”
Revisiting Training Strategies and Generalization Performance in Deep Metric Learning (2020, background): “Generally, performance be- tween criteria is much more similar than literature indicates, as also noted concurrently by Musgrave et al. (2020) and Fehervari et al. (2019).”
Training Vision Transformers for Image Retrieval (2021, unknown): “In particular, we adopt a contrastive loss (Hadsell et al., 2006), which has recently been reafﬁrmed as a very effective metric learning objective (Musgrave et al., 2020; Wang et al., 2020).”
ABO: Dataset and Benchmarks for Real-World 3D Object Understanding (2021, background): “This confirms the risks of saturation and unfair evaluations in existing benchmarks [43] and the need for novel metric learning approaches to handle the large scale and unique challenges of our new benchmark.”
Probabilistic Embeddings for Cross-Modal Retrieval (2021, methodology): “[33] proposed the R-Precision (R-P) metric as an alternative; for every query q, we compute the ratio of positive items in the top-r retrieved items, where r “ |τpqq| is the number of ground-truth matches.”
Fine-Grained Image Analysis With Deep Learning: A Survey (2021, background): “Specifically, deep metric learning [188] attempts to map”
Are We Learning Yet? A Meta Review of Evaluation Failures Across Machine Learning (2021, background): “Even more egregiously, NMI returns higher scores for datasets with more classes, regardless of the algorithm’s performance [101].”