Are Loss Functions All the Same?

By Lorenzo Rosasco, Ernesto De Vito, Andrea Caponnetto, Michele Piana, Alessandro Verri. 2004

527 citations recorded by the source corpus.

In this letter, we investigate the impact of choosing different loss functions from the viewpoint of statistical learning theory. We introduce a convexity assumption, which is met by all loss functions commonly used in the literature, and study how the bound on the estimation error changes with the loss. We also derive a general result on the minimizer of the expected risk for a convex loss function in the case of classification. The main outcome of our analysis is that for classification, the hinge loss appears to be the loss of choice. Other things being equal, the hinge loss leads to a convergence rate practically indistinguishable from the logistic loss rate and much better than the square loss rate. Furthermore, if the hypothesis space is sufficiently rich, the bounds obtained for the hinge loss are not loosened by the thresholding stage.

Subfields: Artificial Intelligence, Computational Mechanics, Optimization & Training, Statistical Learning Theory, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Optimization & Training

In Statistical Learning Theory

In Signal Processing

Citations in the corpus

Towards the Science of Security and Privacy in Machine Learning (2016, methodology): “Examples include the hinge loss [23] used in SVMs or the cross-entropy loss [3].”
Survey on Automated Machine Learning (2019, methodology): “In general, the loss function depends on the underlying problem: a common loss function for regression is the mean squared error (Wang and Bovik, 2009) and hinge loss is often used for classification (Rosasco et al., 2004).”
Data Synthesis based on Generative Adversarial Networks (2018, methodology): “We use the concept of maximum-margin in the hinge loss [31] to control the quality of synthesis process.”
SoK: Security and Privacy in Machine Learning (2018, neutral): “rom X to Y is captured by a function h: X !Y associated with a loss function l h: X Y !R which captures the error made by the prediction h(x) when the true label is y. Examples include the hinge loss [23] used in SVMs or the cross-entropy loss [3]. The learner aims to learn a function h from a famil”
Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons (2018, methodology): “Inspired by this similarity, the alternative loss is designed similar to the hinge loss (Rosasco et al. 2004) used in SVM (Cortes and Vapnik 1995).”
Toward Information Privacy for the Internet of Things: A Nonparametric Learning Approach (2018, methodology): “These include commonly used loss functions like the logistic loss function [45], exponential loss function [46], hinge loss function [47], and quadratic loss function, which have been shown to be computationally efficient and have bounded approximation and estimation errors [44], [48].”
Moving beyond regression techniques in cardiovascular risk prediction: applying machine learning to address analytic challenges (2016, background): “Overall, the above considerations stay the same, though usually the choice of loss functions will change for different outcomes.(39,47)”
Cost-sensitive Multiclass Classification Risk Bounds (2013, background): “In fact, whether this happens is clearly a property of the surrogate loss (Rosasco et al., 2004).”
Minimizing The Misclassification Error Rate Using a Surrogate Convex Loss (2012, background): “For example, based on Zhang (2004), Rosasco et al. (2004) argue that the hinge loss (and also logistic loss) enjoy better rates than other losses like squared loss.”
Robust 1-Bit Compressive Sensing via Binary Stable Embeddings of Sparse Vectors (2011, result): “Previous results in machine learning have shown that for typical convex loss functions, the minimizer of the hinge loss has the tightest bound between expected risk and the Bayes optimal solution [60] and good error rates, especially when considering robustness to outliers [60], [61].”
Deep Learning for Anomaly Detection (2020, background): “nstances randomly sampled from the dataset X, Aand Nrespectively be the pseudo-labeled anomaly and normal instance sets, with X= A∪Nand ∅= A∩N, its loss function is built upon the hinge loss function [131] and can be represented as L query = 1 |X| Õ x∈A,x′∈N max 0,m + f (x′,S;Θ)−f (x,S;Θ) , (25) whe”
On the Learning Dynamics of Deep Neural Networks (2018, background): “, 2014) has shown the resurgence of the hinge loss (Rosasco et al., 2004).”
CoaCor: Code Annotation for Code Retrieval with Reinforcement Learning (2019, background): “c)+cos(v ,v −)) (5) where θdenotes the model parameters, ϵis a constant margin, and vq,vc andvc−are the encoded vectors of Q,C andC−respectively. Essentially, the ranking loss is a kind of hinge loss [46] that promotes the cosine similarity between Q and C to be greater than that between Q andC−by a”
Deep Learning and Its Applications in Biomedicine (2018, background): “However, it often tends to penalize outliers excessively, leading to slower convergence rates [77].”
Joint distribution optimal transportation for domain adaptation (2017, background): “The Hinge loss is better in than mse in 10 out of 12 cases, which is expected given the superiority of the Hinge loss on classification tasks [19].”