Mitigating Neural Network Overconfidence with Logit Normalization

By Hongxin Wei, Renchunzi Xie, Hao Cheng, Lei Feng, Bo An, Yixuan Li. 2022

414 citations recorded by the source corpus.

Detecting out-of-distribution inputs is critical for safe deployment of machine learning models in the real world. However, neural networks are known to suffer from the overconfidence issue, where they produce abnormally high confidence for both in- and out-of-distribution inputs. In this work, we show that this issue can be mitigated through Logit Normalization (LogitNorm) -- a simple fix to the cross-entropy loss -- by enforcing a constant vector norm on the logits in training. Our method is motivated by the analysis that the norm of the logit keeps increasing during training, leading to overconfident output. Our key idea behind LogitNorm is thus to decouple the influence of output's norm during network optimization. Trained with LogitNorm, neural networks produce highly distinguishable confidence scores between in- and out-of-distribution data. Extensive experiments demonstrate the superiority of LogitNorm, reducing the average FPR95 by up to 42.30% on common benchmarks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Ziwei Liu
  2. Yixuan Li
  3. Kaiyang Zhou
  4. Jingkang Yang
  5. Wayne Zhang
  6. Haoqi Wang
  7. Bo Li
  8. Yiyou Sun

In Computer Vision and Pattern Recognition

  1. Ziwei Liu
  2. Luc Van Gool
  3. Chen Change Loy
  4. Xiaogang Wang
  5. Ming-Hsuan Yang
  6. Kaiyang Zhou
  7. Yu Qiao
  8. Hongsheng Li

Citations in the corpus