Regularizing Neural Networks by Penalizing Confident Output Distributions

By Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, Geoffrey E. Hinton. 2017

1,268 citations recorded by the source corpus.

We systematically explore regularizing neural networks by penalizing low entropy output distributions. We show that penalizing low entropy output distributions, which has been shown to improve exploration in reinforcement learning, acts as a strong regularizer in supervised learning. Furthermore, we connect a maximum entropy based confidence penalty to label smoothing through the direction of the KL divergence. We exhaustively evaluate the proposed confidence penalty and label smoothing on 6 common benchmarks: image classification (MNIST and Cifar-10), language modeling (Penn Treebank), machine translation (WMT'14 English-to-German), and speech recognition (TIMIT and WSJ). We find that both label smoothing and the confidence penalty improve state-of-the-art models across benchmarks without modifying existing hyperparameters, suggesting the wide applicability of these regularizers.

Subfields: Artificial Intelligence, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

Citations in the corpus

Semi-Supervised Sequence Modeling with Cross-View Training (2018, methodology): “We apply label smoothing (Szegedy et al., 2016; Pereyra et al., 2017) with a rate of 0.1 to the target labels when training on the labeled data.”
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension (2019, methodology): “During finetuning we use a label smoothed cross entropy loss (Pereyra et al., 2017), with the smoothing parameter set to 0.”
Scaling Neural Machine Translation (2018, background): “1 weight for the uniform prior distri-2 numrefs.1+smooth.exp+test.wmt14/full+tok.13a+ version.1.2.9 bution over the vocabulary (Szegedy et al., 2015; Pereyra et al., 2017).”
Beyond English-Centric Multilingual Machine Translation (2020, methodology): “We train with the Adam optimizer (Kingma and Ba, 2015) and warmup first for 4000 updates, with label smoothing 0.1 (Szegedy et al., 2015; Pereyra et al., 2017).”
mixup: Beyond Empirical Risk Minimization (2017, background): “Recent approaches have also proposed to regularize the output distribution of a neural network by label smoothing (Szegedy et al., 2016), or penalizing high-conﬁdence softmax distributions (Pereyra et al., 2017).”
Pay Less Attention with Lightweight and Dynamic Convolutions (2019, background): “1 weight for the uniform prior distribution over the vocabulary (Szegedy et al., 2015; Pereyra et al., 2017).”
Categorical Reparameterization with Gumbel-Softmax (2016, background): “The practical outcome of this paper is a simple, differentiable approximate sampling mechanism for categorical variables that can be integrated into neural networks and trained using standard back-propagation.”
On Calibration of Modern Neural Networks (2017, background): “Pereyra et al. (2017) penalize overconfident predictions as a form of regularization.”
Pre-trained language model representations for language generation (2019, background): “1 (Szegedy et al., 2015; Pereyra et al., 2017).”
Large scale distributed neural network training through online distillation (2018, neutral): “One concern would be that codistillation is merely a way of penalizing confident output distributions (Pereyra et al., 2017) or smoothing the labels, so we also compared to two label smoothing baselines.”
CPR: Classifier-Projection Regularization for Continual Learning (2020, methodology): “Note that, for example, when g is uniform distribution PU in ∆M , the regularization term corresponds to entropy maximization proposed in Pereyra et al. (2017), and when g is another classifier’s output fθ′ , Eq.”
Excessive Invariance Causes Adversarial Vulnerability (2018, background): “Furthermore, extended cross-entropy losses via entropy terms (Pereyra et al., 2017) or minimizing predictability of variables (Schmidhuber, 1991) has been introduced for other applications.”
Reliable Uncertainty Estimates in Deep Neural Networks using Noise Contrastive Priors (2018, background): “Output regularization is also a classic idea from the maximum entropy principle (Jaynes, 1957), where it has motivated label smoothing (Szegedy et al., 2016) and entropy penalties (Pereyra et al., 2017).”
Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples (2017, background): “We remark that minimizing a similar KL loss was studied recently for different purposes (Lee et al., 2017; Pereyra et al., 2017).”
Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks (2017, methodology): “We observe that after using temperature scaling in the softmax function (Hinton et al., 2015; Pereyra et al., 2017) and adding small controlled perturbations to inputs, calibrate the prediction conﬁdence in classiﬁcation tasks (Guo et al., 2017).”
Measuring and regularizing networks in function space (2018, background): “It may be possible, for example, to use the norm to regularize the conﬁdence of the output function (e.g. Pereyra et al. (2017)).”
Deep Mutual Learning (2017, methodology): “ers due to being in wide valleys rather than narrow crevices [4, 10] – so that small perturbations do not change the prediction efficacy drastically; and that deep networks are better than might be expected at finding these good solutions [30], but that the tendency towards finding robust minima can”
ESPnet: End-to-End Speech Processing Toolkit (2018, methodology): “We implemented unigram smoothing, where the distribution of remaining labels is set to be proportional to the unigram distribution of the labels [40].”