Adversarial Robustness vs. Model Compression, or Both?

By Shaokai Ye, Xue Lin, Kaidi Xu, Sijia Liu, Hao Cheng, Jan-Henrik Lambrechts, Huan Zhang, Aojun Zhou, Kaisheng Ma, Yanzhi Wang. 2019

181 citations recorded by the source corpus.

It is well known that deep neural networks (DNNs) are vulnerable to adversarial attacks, which are implemented by adding crafted perturbations onto benign examples. Min-max robust optimization based adversarial training can provide a notion of security against adversarial attacks. However, adversarial robustness requires a significantly larger capacity of the network than that for the natural training with only benign examples. This paper proposes a framework of concurrent adversarial training and weight pruning that enables model compression while still preserving the adversarial robustness and essentially tackles the dilemma of adversarial training. Furthermore, this work studies two hypotheses about weight pruning in the conventional setting and finds that weight pruning is essential for reducing the network model size in the adversarial setting; training a small model from scratch even with inherited initialization from the large model cannot achieve neither adversarial robustness nor high standard accuracy. Code is available at https://github.com/yeshaokai/Robustness-Aware-Pruning-ADMM.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In Safety, Robustness & Interpretability

Citations in the corpus

Towards Robust Vision Transformer (2021, background): “…strengthen it in different ways, e.g., stronger data augmentation [16, 18, 33], carefully designed [36,44] or searched [9,13] network architecture, improved training strategy [22, 26, 47], quantization [24] and pruning [49] of the weights, better pooling [41,53] or activation functions [46], etc.”
MedViT: A Robust Vision Transformer for Generalized Medical Image Classification (2023, background): “36], strong data augmentation [37, 38], searched network architecture [39, 40], improved training strategy [41–43], pruning [44] of the weights, and quantization [45], activation functions [46] or better pooling [47, 48].”
Task-Oriented Feature Distillation (2020, methodology): “Various techniques have been proposed to tackle this issue, including pruning [15, 41, 72, 67, 39], quantization [35, 46, 11, 50], lightweight model de-sign [23, 49, 27], and knowledge distillation (KD) [22, 53, 70].”
Self-Distillation: Towards Efficient and Compact Neural Networks (2021, methodology): “Typical approaches include pruning [12], [13], [14], [15], [16], quantization [17], [18], [19], [20], light-weight neural network design [21], [22], [23], [24], [25], low rank factorization [26], [27] and knowledge distillation [28], [29], [30], [31].”
Advances in adversarial attacks and defenses in computer vision: A survey (2021, background): “[346] proposed a concurrent adversarial training and weight pruning strategy to address this specific issue.”
Triple Wins: Boosting Accuracy, Robustness and Efficiency Together by Enabling Input-Adaptive Inference (2020, background): “Two latest works (Ye et al., 2019; Gui et al., 2019) examined the robustness of compressed models, and concluded similar observations that the relationship between mode size and robustness depends on compression methods and are often non-monotonic.”