Robust Bi-Tempered Logistic Loss Based on Bregman Divergences

By Ehsan Amid, Manfred K. Warmuth, Rohan Anil, Tomer Koren. 2019

147 citations recorded by the source corpus.

We introduce a temperature into the exponential function and replace the softmax output layer of neural nets by a high temperature generalization. Similarly, the logarithm in the log loss we use for training is replaced by a low temperature logarithm. By tuning the two temperatures we create loss functions that are non-convex already in the single layer case. When replacing the last layer of the neural nets by our bi-temperature generalization of logistic loss, the training becomes more robust to noise. We visualize the effect of tuning the two temperatures in a simple setting and show the efficacy of our method on large data sets. Our methodology is based on Bregman divergences and is superior to a related two-temperature method using the Tsallis divergence.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Tempered Sigmoid Activations for Deep Learning with Differential Privacy (2020, methodology): “…parameter T in Equation (3) can be used as a knob to control the norm of the gradient of the loss function, and with an appropriate choice avoids these 1We experimented with the tempered loss of (Amid et al. 2019) but did not find any improvements for DP-SGD training. two issues of clipping.”
Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations (2021, background): “, 2020), re-weight the loss to encourage deep neural nets to fit on correct labels (Liu & Tao, 2015), propose robust loss functions (Natarajan et al., 2013; Ghosh et al., 2017; Zhang & Sabuncu, 2018; Amid et al., 2019; Wang et al., 2019; Liu & Guo, 2020), or introduce a robust regularizer (Liu et al”
Does label smoothing mitigate label noise? (2020, background): “Indeed, label smoothing is conspicuously absent in most treatments of the noisy label problem (Patrini et al., 2016; Han et al., 2018b; Charoenphakdee et al., 2019; Thulasidasan et al., 2019; Amid et al., 2019; Menon et al., 2020).”
Large-Scale Differentially Private BERT (2021, background): “…et al., 2020), more efﬁcient architectures such as MLP-Mixer, and F-NETs (Tolstikhin et al., 2021; Lee-Thorp et al., 2021) for longer sequences, multi-step training at mega-batch sizes (Choi et al., 2019; Agarwal et al., 2020), loss functions that are robust to label noise (Amid et al., 2019).”
Characterizing Private Clipped Gradient Descent on Convex Generalized Linear Problems (2020, unknown): “], Such loss functions appear commonly in robust regression, such as Savage loss [28 Tan-gent loss [27], and tempered loss [3].”