On Loss Functions for Deep Neural Networks in Classification

By Katarzyna Janocha, Wojciech M. Czarnecki. arXiv.org, 2017

610 citations recorded by the source corpus.

Deep neural networks are currently among the most commonly used classifiers. Despite easily achieving very good performance, one of the best selling points of these models is their modular design - one can conveniently adapt their architecture to specific needs, change connectivity patterns, attach specialised layers, experiment with a large amount of activation functions, normalisation schemes and many others. While one can find impressively wide spread of various configurations of almost every aspect of the deep nets, one element is, in authors' opinion, underrepresented - while solving classification problems, vast majority of papers and applications simply use log loss. In this paper we try to investigate how particular choices of loss functions affect deep models and their learning dynamics, as well as resulting classifiers robustness to various effects. We perform experiments on classical datasets, as well as provide some additional, theoretical insights into the problem. In particular we show that L1 and L2 losses are, quite surprisingly, justified classification objectives for deep nets, by providing probabilistic interpretation in terms of expected misclassification. We also introduce two losses which are not typically used as deep nets objectives and show that they are viable alternatives to the existing ones.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/d589392ab606a3d2861988ebcba95176517939ec

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

A Deep Collocation Method for the Bending Analysis of Kirchhoff Plate (2021, background): “For the approximation defined by f (x; θ), in order to find the weights and biases, a loss function L (f, w) is defined to be minimised [56].”
A Framework for Evaluating Gradient Leakage Attacks in Federated Learning (2020, background): “[16] has studied the impact of different distance functions on training efficiency.”
Convolutional Neural Networks for Classification of Alzheimer's Disease: Overview and Reproducible Evaluation (2019, background): “Other loss functions were also discussed in the literature, such as mean squared error (MSE) loss and hinge loss (see (Janocha and Czarnecki, 2017) for details).”
State-of-the-art in artificial neural network applications: A survey (2018, methodology): “In modeling on loss function, a paper by Janocha, and Czarnecki [75], investigated the effect of loss functions choices on deep models and its learning dynamics, and the resulting classifiers robustness to different effects.”
Reducing the Hausdorff Distance in Medical Image Segmentation With Convolutional Neural Networks (2019, methodology): “In this experiment, we replaced the squared difference term (p − q)2 in the DT-based loss function (Equation (8)) with some of the alternatives proposed in [33], [47], [48].”
Deep Autoencoder based Energy Method for the Bending, Vibration, and Buckling Analysis of Kirchhoff Plates (2020, background): “In order to ﬁnd the hyperparameters of the deep autoencoder including weights and biases, a loss function L ( f , w ) is constructed [48].”
Why Do Better Loss Functions Lead to Less Transferable Features? (2020, background): “Janocha and Czarnecki [32] previously studied both empirical and mathematical similarities and differences among loss functions for supervised classification in the context of deep learning.”
Hands-On Bayesian Neural Networks—A Tutorial for Deep Learning Users (2020, background): “Covering these metrics is out of the scope of this tutorial, for more details the reader can refer to [35].”