A Unified Gradient Regularization Family for Adversarial Examples

By Chunchuan Lyu, Kaizhu Huang, Hai-Ning Liang. 2015

221 citations recorded by the source corpus.

Adversarial examples are augmented data points generated by imperceptible perturbation of input samples. They have recently drawn much attention with the machine learning and data mining community. Being difficult to distinguish from real examples, such adversarial examples could change the prediction of many of the best learning models including the state-of-the-art deep learning models. Recent attempts have been made to build robust models that take into account adversarial examples. However, these methods can either lead to performance drops or lack mathematical motivations. In this paper, we propose a unified framework to build robust machine learning models against adversarial examples. More specifically, using the unified framework, we develop a family of gradient regularization methods that effectively penalize the gradient of loss function w.r.t. inputs. Our proposed framework is appealing in that it offers a unified view to deal with adversarial examples. It incorporates another recently-proposed perturbation based approach as a special case. In addition, we present some visual effects that reveals semantic meaning in those perturbations, and thus support our regularization method and provide another explanation for generalizability of adversarial examples. By applying this technique to Maxout networks, we conduct a series of experiments and achieve encouraging results on two benchmark datasets. In particular, we attain the best accuracy on MNIST data (without data augmentation) and competitive performance on CIFAR-10 data.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Towards the Science of Security and Privacy in Machine Learning (2016, neutral): “to other gradient-based penalties in [82], [83].”
Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning (2017, background): “Hybrid approaches based on regularizing gradients through simulation of the corresponding attacks have also been recently proposed to improve the security of deep networks to evasion attacks [90, 91].”
Adversarial Vulnerability of Neural Networks Increases With Input Dimension (2018, background): “Lyu et al. (2015) discuss and use the connection between gradient-penalties and adversarial augmentation, but never actually compare both in experiments.”
Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey (2018, methodology): “[28] also used the notion of penalizing the gradient of loss function of network models with respect to the inputs to incorporate robustness in the net-”
Why Do Adversarial Attacks Transfer? Explaining Transferability of Evasion and Poisoning Attacks (2018, background): “, via regularization, may allow us to learn more robust classifiers not only against evasion [22, 35, 39, 44] but also against poisoning availability attacks.”
A survey on adversarial attacks and defences (2021, background): “The primary objective of the adversarial training is to increase model robustness by injecting adversarial examples into the training set [14,18,70,71].”
Unifying Adversarial Training Algorithms with Flexible Deep Data Gradient Regularization (2016, methodology): “Close to our work, (Lyu et al., 2015) present a heuristic way to optimize a special case of this objective.”
Robustness via Curvature Regularization, and Vice Versa (2018, background): “We finally note that prior works have attempted to improve the robustness using gradient regularization [10, 15, 19].”
A Kernel Perspective for Regularizing Deep Neural Networks (2018, unknown): “…the context of robust optimization, for regularization or robustness, noting that a penalty based on the gradient of the loss function x 7→ ℓ ( y, f ( x )) can give a good approximation of (7) when ǫ is small (Drucker & Le Cun, 1991; Lyu et al., 2015; Roth et al., 2018; Simon-Gabriel et al., 2019).”
On the Effectiveness of Mitigating Data Poisoning Attacks with Gradient Shaping (2020, background): “Gradient Regularization as a Defense Prior work, especially in the context of neural networks, has proposed imposing regularization penalties on a model’s gradients to improve the accuracy [16], interpretability [33] or robustness against adverarial examples [27].”
Unifying Adversarial Training Algorithms with Data Gradient Regularization (2017, background): “In an approach that ends up closely related to ours, Lyu et al. (2015) consider the objective minθ maxr:||r||p≤σ L(x + r; θ ) and a linearized inner version maxr:||r||p≤σ L(x) + ∇xL Tr.”
Regularization for Deep Learning: A Taxonomy (2017, background): “Loss-invariant backpropagation (Demyanov et al., 2015, Sec. 3.1; Lyu et al., 2015) (L(2)) norm of gradient of loss w.”
First-Order Adversarial Vulnerability of Neural Networks and Input Dimension (2018, background): “Lyu et al. (2015) discuss and use the connection between gradient-penalties and adversarial augmentation, but surprisingly never empirically compare both, which we do in Section 4.1.”
Adversarial Weight Perturbation Helps Robust Generalization (2020, methodology): “AT explicitly ﬂattens the input loss landscape by training on adversarially perturbed examples, while there are other methods doing this by gradient regularization [23, 38], curvature regularization [27], and local linearity regularization [36].”