Loss Functions for Top-k Error: Analysis and Insights

By Maksim Lapin, Matthias Hein, Bernt Schiele. 2015

110 citations recorded by the source corpus.

In order to push the performance on realistic computer vision tasks, the number of classes in modern benchmark datasets has significantly increased in recent years. This increase in the number of classes comes along with increased ambiguity between the class labels, raising the question if top-1 error is the right performance measure. In this paper, we provide an extensive comparison and evaluation of established multiclass methods comparing their top-k performance both from a practical as well as from a theoretical perspective. Moreover, we introduce novel top-k loss functions as modifications of the softmax and the multiclass SVM losses and provide efficient optimization schemes for them. In the experiments, we compare on various datasets all of the proposed and established methods for top-k error optimization. An interesting insight of this paper is that the softmax loss yields competitive top-k performance for all k simultaneously. For a specific top-k error, our new top-k losses lead typically to further improvements while being faster to train than the softmax.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Why ReLU Networks Yield High-Confidence Predictions Far Away From the Training Data and How to Mitigate the Problem (2018, background): “Please note that from the point of view of Bayesian decision theory the softmax function is the correct transfer function [19] for the cross-entropy loss turning the classiﬁer output for the conditional probability at x .”
Training Neural Networks for and by Interpolation (2019, methodology): “We use the top-k truncated cross-entropy (Lapin et al., 2016) as our loss function for training the model on ImageNet.”
Top-K Off-Policy Correction for a REINFORCE Recommender System (2018, unknown): “igure3shows the policies � �learned with the standard (left) and top-�off-policy correction (right). We can see that with the standard off-policy correction, although the learned policy is calibrated [23] in the sense that it still maintains the ordering of items Figure 3: Learned policy � � (left):”
Video Visual Relation Detection (2017, methodology): “Since we are only interested in the top relation prediction scores, we use softmax loss which has recently been proved to be e ective in this case, both theoretically and empirically [13, 42].”
Optimizing the Dice Score and Jaccard Index for Medical Image Segmentation: Theory and Practice (2019, background): “For the Hamming similarity, cross-entropy loss and other convex surrogates are statistically consistent [4,13].”
Fast Differentiable Sorting and Ranking (2020, background): “While several works opt to approximate ranking metrics directly (Chapelle & Wu, 2010; Adams & Zemel, 2011; Lapin et al., 2016), others introduce “soft” ranks, which can then be plugged into any differentiable loss function.”
Towards a Visual Privacy Advisor: Understanding and Predicting Privacy Risks in Images (2017, unknown): “In a pilot study, we found that the multilabel SVM with smoothed hinge loss [25] yields better results than SVM multi-label prediction [7] and cross-entropy loss.”