Defensive Quantization: When Efficiency Meets Robustness

By Ji Lin, Chuang Gan, Song Han. 2019

222 citations recorded by the source corpus.

Neural network quantization is becoming an industry standard to efficiently deploy deep learning models on hardware platforms, such as CPU, GPU, TPU, and FPGAs. However, we observe that the conventional quantization approaches are vulnerable to adversarial attacks. This paper aims to raise people's awareness about the security of the quantized models, and we designed a novel quantization methodology to jointly optimize the efficiency and robustness of deep learning models. We first conduct an empirical study to show that vanilla quantization suffers more from adversarial attacks. We observe that the inferior robustness comes from the error amplification effect, where the quantization operation further enlarges the distance caused by amplified noise. Then we propose a novel Defensive Quantization (DQ) method by controlling the Lipschitz constant of the network during quantization, such that the magnitude of the adversarial noise remains non-expansive during inference. Extensive experiments on CIFAR-10 and SVHN datasets demonstrate that our new quantization method can defend neural networks against adversarial examples, and even achieves superior robustness than their full-precision counterparts while maintaining the same hardware efficiency as vanilla quantization approaches. As a by-product, DQ can also improve the accuracy of quantized models without adversarial attack.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiation, Efficiency & Compression, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiation

In Efficiency & Compression

In Safety, Robustness & Interpretability

Citations in the corpus

Adversarial Training for Free! (2019, background): “Examples include Adversarial Logit Pairing (ALP) [Kannan et al., 2018], Feature Denoising [Xie et al., 2019], Defensive Quantization [Lin et al., 2019], Thermometer Encoding [Buckman et al., 2018], PixelDefend [Song et al., 2017], Robust Manifold Defense [Ilyas et al., 2017], L2-nonexpansive nets…”
You Only Propagate Once: Painless Adversarial Training Using Maximal Principle (2019, methodology): “To improve the robustness of neural networks to adversarial examples, many defense approaches have been proposed, like, adversarial trainining [5], orthogonal regularization [10,11], L2 nonexpansive networks [12], Bayesian method [13], random masking [14], TRADES [7], rejecting adversarial examples ”
Adversarial T-Shirt! Evading Person Detectors in a Physical World (2019, background): “The vulnerability of deep neural networks (DNNs) against adversarial attacks (namely, perturbed inputs deceiving DNNs) has been found in applications spanning from image classification to speech recognition [33,21,34,37,6,32,2].”
Stochastic Precision Ensemble: Self-Knowledge Distillation for Quantized Deep Neural Networks (2020, unknown): “Activation quantized models are known to be vulnerable to the adversarial noise (Lin, Gan, and Han 2019).”
When NAS Meets Robustness: In Search of Robust Architectures Against Adversarial Attacks (2019, background): “Extensive efforts have been proposed to enhance the robustness, including preprocessing techniques [3, 28, 38], feature denoising [35], regularization [41, 13, 17], adding unlabeled data [6, 29], model ensemble [33, 24], where adversarial training [9, 21] turns out to be the most effective and stand”
Adversarially Robust Distillation (2019, background): “Defensive Quantization involves quantizing a network while minimizing the Lipschitz constant to encourage robustness [14].”
Triple Wins: Boosting Accuracy, Robustness and Efficiency Together by Enabling Input-Adaptive Inference (2020, background): “Lin et al. (2019) found that activation quantization may hurt robustness, but can be turned into effective defense if enforcing continuity constraints.”
Advances in adversarial attacks and defenses in computer vision: A survey (2021, background): “Another example of attacks on quantized networks can be found in [196].”
You Only Propagate Once: Accelerating Adversarial Training via Maximal Principle (2019, methodology): “To improve the robustness of neural networks to adversarial examples, many defense strategies and models have been proposed, such as adversarial training [25], orthogonal regularization [6,22], L2 nonexpansive networks [28], Bayesian method [41], random masking [24], TRADES [42], rejecting adversari”
Bit-Flip Attack: Crushing Neural Network With Progressive Bit Search (2019, background): “Thus some of the popular adversarial defense methods [26, 23] fail against our BFA attack.”