Learning Deep Embeddings with Histogram Loss

By Evgeniya Ustinova, Victor S. Lempitsky. 2016

368 citations recorded by the source corpus.

We suggest a new loss for learning deep embeddings. The key characteristics of the new loss is the absence of tunable parameters and very good results obtained across a range of datasets and problems. The loss is computed by estimating two distribution of similarities for positive (matching) and negative (non-matching) point pairs, and then computing the probability of a positive pair to have a lower similarity score than a negative pair based on these probability estimates. We show that these operations can be performed in a simple and piecewise-differentiable manner using 1D histograms with soft assignment operations. This makes the proposed loss suitable for learning deep embeddings using stochastic optimization. The experiments reveal favourable results compared to recently proposed loss functions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning Spread-Out Local Feature Descriptors (2017, background): “The main problem with the pairwise loss is that the margin parameters are often difficult to choose [25].”
Learning Deep Disentangled Embeddings with the F-Statistic Loss (2018, methodology): “For this task, we evaluate using two datasets—CUHK03 (Li et al., 2014) and Market-1501 (Zheng et al., 2015)—following the methodology of Ustinova & Lempitsky (2016).”
Sampling Matters in Deep Embedding Learning (2017, background): “Deep networks trained to respect pairwise relationships have emerged as the most successful embedding models [4, 6, 25, 34].”
R2D2: Repeatable and Reliable Detector and Descriptor (2019, methodology): “Inspired by recent advances in listwise losses [18, 58], He et al .”
Class Rectification Hard Mining for Imbalanced Deep Learning (2017, background): “In the spirit of [45], we represent the distribution of positive P + and negative P − pair sets with histograms where ς i,j,t = otherwise (13) and ∆ deﬁnes the step between two adjacent bins.”
Person Re-identification by Deep Learning Multi-scale Representations (2017, neutral): “3 - HL [56] 59.”
Deep Metric Learning: A Survey (2019, methodology): “Histogram Loss [84] Easy sampling Image recognition Image retrieval Person re-ID CUB-200-2011 [68] Online Products [70] CUHK03 [72] Market-1501 [71] aims the distributions of the similarities of less overlapping positive and negative pairs.”
Learning Beyond Human Expertise with Generative Models for Dental Restorations (2018, neutral): “More recently, several works [44,45] have shown successes in incorporating histogram learning into deep neural networks for classiﬁcation tasks and feature embedding.”
Mining on Manifolds: Metric Learning Without Labels (2018, background): “Of course, given the positive and negative pools, there are many more possibilities in sampling positives and negatives and forming losses that are functions of more than two or three items [29, 36, 5, 27, 59, 67].”
Metric Learning for Novelty and Anomaly Detection (2018, methodology): “In practice, we have not implemented a two-branches Siamese network but followed recent works [18, 28] which devise a more efficient approach to minimize losses traditionally computed with Siamese networks.”
Deep Metric Learning with Angular Loss (2017, background): “Nevertheless, all above-mentioned losses are deﬁned in term of distances of points, and very few [31] has considered other possible forms of loss.”
Smart Mining for Deep Metric Learning (2017, background): “Ustinova and Lempitsky [22] presented a loss function that minimises the integral of the product between the distribution of negative similarities and the cumulative density function of the positive similarities.”
Escape from Cells: Deep Kd-Networks for the Recognition of 3D Point Cloud Models (2017, methodology): “A depth-15 Kd-network trained with the histogram loss [33] was used for this task with leaf representation of size 16 (obtained from the three coordinates using an additional multiplicative layer) and intermediate representations of sizes 32−32−64−64−128−128− 256−256−512−512−1024−1024−2048−2048−512.”
Heated-Up Softmax Embedding (2018, background): “Designing structured losses to consider all the possible training pairs or triplets within one training batch and perform “soft” hard mining can be an alternative solution for hard mining (Song et al., 2017; 2016; Ustinova & Lempitsky, 2016).”
Semantic Search in Millions of Equations (2020, methodology): “The triplet loss [2] we have previously used [20], contrasts the similarity between a positive pair of examples and a negative pair of examples and demands that the similar pair has a higher similarity by a user-defined margin ∆, usually set to 1. ℓt (x ,x+,x−) = max(0,∆ − ⟨ϕ̄(x), ϕ̄(x+)⟩ + ⟨ϕ̄(x), ”
Revisiting Training Strategies and Generalization Performance in Deep Metric Learning (2020, background): “Further, rankingbased objectives are also extended to histogram matching, as proposed in (Ustinova & Lempitsky, 2016).”
Adapted Deep Embeddings: A Synthesis of Methods for k-Shot Inductive Transfer Learning (2018, methodology): “1 Various objective functions have been proposed for deep metric learning, all of which aim to ensure that instances of the same class are near one another in the embedding space and instances of different classes are far apart [2, 4, 18, 20, 21, 26, 28, 32, 34, 35, 38].”