Feature Purification: How Adversarial Training Performs Robust Deep Learning

By Zeyuan Allen-Zhu, Yuanzhi Li. 2020

177 citations recorded by the source corpus.

Despite the empirical success of using adversarial training to defend deep learning models against adversarial perturbations, so far, it still remains rather unclear what the principles are behind the existence of adversarial perturbations, and what adversarial training does to the neural network to remove them. In this paper, we present a principle that we call feature purification, where we show one of the causes of the existence of adversarial examples is the accumulation of certain small dense mixtures in the hidden weights during the training process of a neural network; and more importantly, one of the goals of adversarial training is to remove such mixtures to purify hidden weights. We present both experiments on the CIFAR-10 dataset to illustrate this principle, and a theoretical result proving that for certain natural classification tasks, training a two-layer neural network with ReLU activation using randomly initialized gradient descent indeed satisfies this principle. Technically, we give, to the best of our knowledge, the first re-sult proving that the following two can hold simultaneously for training a neural network with ReLU activation. (1) Training over the original data is indeed non-robust to small adversarial perturbations of some radius. (2) Adversarial training, even with an empirical perturbation algorithm such as FGM, can in fact be provably robust against any perturbations of the same radius. Finally, we also prove a complexity lower bound, showing that low complexity models such as linear classifiers, low-degree polynomials, or even the neural tangent kernel for this network, cannot defend against perturbations of this same radius, no matter what algorithms are used to train them. 11The full version of this paper can be found at https://arxiv.org/abs/-2005.l0190.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

In Safety, Robustness & Interpretability

Citations in the corpus

LoRA: Low-Rank Adaptation of Large Language Models (2021, background): “theoretical result in [4] suggests that low-rank adaptations can be useful for adversarial training.”
Do Adversarially Robust ImageNet Models Transfer Better? (2020, background): “Understanding why robust networks transfer particularly well remains an open problem, likely relating to prior work that analyses the features these networks use [Eng+19a; Sha+19; AL20].”
Do Wider Neural Networks Really Help Adversarial Robustness? (2020, background): “[2] showed that the adversarial examples stem from the accumulation of small dense mixtures in the hidden weights during training and adversarial training works by removing such mixtures.”
Learning Over-Parametrized Two-Layer ReLU Neural Networks beyond NTK (2020, background): “Allen-Zhu and Li (2019a, 2020a) show that over-parametrized neural networks can learn certain concept class more efficient than any kernel method.”