When NAS Meets Robustness: In Search of Robust Architectures Against Adversarial Attacks

By Minghao Guo, Yuzhe Yang, Rui Xu, Ziwei Liu. 2019

178 citations recorded by the source corpus.

Recent advances in adversarial attacks uncover the intrinsic vulnerability of modern deep neural networks. Since then, extensive efforts have been devoted to enhancing the robustness of deep networks via specialized learning algorithms and loss functions. In this work, we take an architectural perspective and investigate the patterns of network architectures that are resilient to adversarial attacks. To obtain the large number of networks needed for this study, we adopt one-shot neural architecture search, training a large network for once and then finetuning the sub-networks sampled therefrom. The sampled architectures together with the accuracies they achieve provide a rich basis for our study. Our ''robust architecture Odyssey'' reveals several valuable observations: 1) densely connected patterns result in improved robustness; 2) under computational budget, adding convolution operations to direct connection edge is effective; 3) flow of solution procedure (FSP) matrix is a good indicator of network robustness. Based on these observations, we discover a family of robust architectures (RobNets). On various datasets, including CIFAR, SVHN, Tiny-ImageNet, and ImageNet, RobNets exhibit superior robustness performance to other widely used architectures. Notably, RobNets substantially improve the robust accuracy (~5% absolute gains) under both white-box and black-box attacks, even with fewer parameter numbers. Code is available at https://github.com/gmh14/RobNets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Alan L. Yuille
2. Jun Zhu
3. Quoc V. Le
4. Hang Su
5. Bernt Schiele
6. Mubarak Shah
7. Boqing Gong
8. Ajmal Saeed Mian

In Computer Vision and Pattern Recognition

1. Alan L. Yuille
2. Luc Van Gool
3. Dacheng Tao
4. Bernt Schiele
5. Trevor Darrell
6. Xiaogang Wang
7. Ming-Hsuan Yang
8. Andrew Zisserman

In Safety, Robustness & Interpretability

1. Tianyu Pang
2. Hang Su
3. Jun Zhu
4. Yinpeng Dong
5. Ajmal Saeed Mian
6. Naveed Akhtar
7. Alan L. Yuille
8. Xiao Yang

Citations in the corpus

• Towards Robust Vision Transformer (2021, background): “, stronger data augmentation [16, 18, 33], carefully designed [36,44] or searched [9,13] network architecture, improved training strategy [22, 26, 47], quantization [24] and pruning [49] of the weights, better pooling [41, 53] or activation functions [46], etc.”
• A Comprehensive Survey of Neural Architecture Search (2020, methodology): “In addition, RobNet [150] proposes to use the NAS method to generate a large number of network architectures by analyzing the differences between strong and poorly performing architectures to find out which network structures help to obtain a robust network architecture.”
• MedViT: A Robust Vision Transformer for Generalized Medical Image Classification (2023, background): “36], strong data augmentation [37, 38], searched network architecture [39, 40], improved training strategy [41–43], pruning [44] of the weights, and quantization [45], activation functions [46] or better pooling [47, 48].”
• Adversarially Robust Neural Architectures (2020, unknown): “%) over that of PC-DARTS under PGD100 attack.”
• Advances in adversarial attacks and defenses in computer vision: A survey (2021, background): “[366] also proposed RobNets, designed with neural architecture search, which are claimed to provide up to 5% gain in robust accuracy on large datasets, e.”
• Smooth Adversarial Training (2020, result): “Our results also corroborate the recent findings that there exist certain network architectures which have better adversarial robustness [8, 21, 74].”
• Cross-Entropy Loss Functions: Theoretical Analysis and Applications (2023, background): “, 2021a;b), neural network architectures (Xie & Yuille, 2020; Xie et al., 2019; Liu et al., 2020; Guo et al., 2020), weight averaging/perturbation (Gowal et al.”
• A Closer Look at Accuracy vs. Robustness (2020, methodology): “We believe that in order to achieve both robustness and accuracy, future work may need to redesign other aspects of the training process such as better network architectures using neural architecture search [13, 19, 47, 69].”
• Bag of Tricks for Adversarial Training (2020, methodology): “Recently, Guo et al. (2020) blend in the technique of AutoML to explore robust architectures.”