Label Smoothing and Logit Squeezing: A Replacement for Adversarial Training?

By Ali Shafahi, Amin Ghiasi, Furong Huang, Tom Goldstein. 2019

44 citations recorded by the source corpus.

Adversarial training is one of the strongest defenses against adversarial attacks, but it requires adversarial examples to be generated for every mini-batch during optimization. The expense of producing these examples during training often precludes adversarial training from use on complex image datasets. In this study, we explore the mechanisms by which adversarial training improves classifier robustness, and show that these mechanisms can be effectively mimicked using simple regularization methods, including label smoothing and logit squeezing. Remarkably, using these simple regularization methods in combination with Gaussian noise injection, we are able to achieve strong adversarial robustness -- often exceeding that of adversarial training -- using no adversarial examples.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tianyu Pang
2. Hang Su
3. Jun Zhu
4. Yinpeng Dong
5. Xiao Yang
6. Tom Goldstein
7. Mahyar Najibi
8. Zheng Xu

In Electrical and Electronic Engineering

1. Tianyu Pang
2. Jun Zhu
3. Zheng Xu
4. Larry S. Davis
5. Christoph Studer
6. Tom Goldstein
7. Hang Su
8. Xiao Yang

Citations in the corpus

• Adversarial Training for Free! (2019, background): “Another popular defense strategy is to regularize the training loss using label smoothing, logit squeezing, or a Jacobian regularization [Shafahi et al., 2019a, Mosbach et al., 2018, Ross and Doshi-Velez, 2018, Hein and Andriushchenko, 2017, Jakubovitz and Giryes, 2018, Yu et al., 2018].”
• Bag of Tricks for Adversarial Training (2020, methodology): “…other hand, due to the high computational cost of AT, many efforts are devoted to accelerating the training procedure via reusing the computations (Shafahi et al., 2019b; Zhang et al., 2019a), adaptive adversarial steps (Wang et al., 2019; Zhang et al., 2020) or one-step training (Wong et al.,…”