Improving the Robustness of Deep Neural Networks via Stability Training

By Stephan Zheng, Yang Song, Thomas K. Leung, Ian J. Goodfellow. 2016

678 citations recorded by the source corpus.

In this paper we address the issue of output instability of deep neural networks: small perturbations in the visual input can significantly distort the feature embeddings and output of a neural network. Such instability affects many deep architectures with state-of-the-art performance on a wide range of computer vision tasks. We present a general stability training method to stabilize deep networks against small input distortions that result from various types of common image processing, such as compression, rescaling, and cropping. We validate our method by stabilizing the state of-the-art Inception architecture [11] against these types of distortions. In addition, we demonstrate that our stabilized model gives robust state-of-the-art performance on largescale near-duplicate detection, similar-image ranking, and classification on noisy datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

Do Image Classifiers Generalize Across Time? (2019, background): “[31] only provide anecdotal evidence from a small number of videos.”
Benchmarking Neural Network Robustness to Common Corruptions and Perturbations (2019, background): “Zheng et al. (2016) also ﬁnd that ﬁne-tuning on noisy images can cause underﬁtting, so they encourage the noisy image softmax distribution to match the clean image softmax.”
Scaling Laws for Reward Model Overoptimization (2022, methodology): “…of adversarial examples in all kinds of neural networks [Szegedy et al., 2013, Lin et al., 2017, Ebrahimi et al., 2018, Dai et al., 2018], and proposed methods to measure and increase neural network robustness [Gu and Rigazio, 2014, Zheng et al., 2016, Carlini et al., 2019, Guo et al., 2021].”
On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima (2016, background): “Similarly, we received no generalization benefit in stability training proposed by [44].”
AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty (2019, background): “The Jensen-Shannon Consistency Loss impels to model to be stable, consistent, and insensitive across a diverse range of inputs (Bachman et al., 2014; Zheng et al., 2016; Kannan et al., 2018).”
Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning (2017, background): “, training neural networks with bounded activation functions and training data augmentation to improve stability to input changes [112, 113].”
On Detecting Adversarial Perturbations (2017, background): “In contrast, Zheng et al. (2016) propose to append a stability term to the objective function, which forces the model to have similar outputs for samples of the training set and their perturbed versions.”
Certified Adversarial Robustness via Randomized Smoothing (2019, methodology): “We did not experiment with stability training (Zheng et al., 2016), as was proposed in Li et al. (2018).”
Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods (2017, background): “There has been a significant amount of work studying methods to construct adversarial examples [5, 8, 13, 26, 30, 39] and to make networks robust against adversarial examples [3, 16, 20, 21, 31, 33, 34, 41].”
DeepXplore: Automated Whitebox Testing of Deep Learning Systems (2017, background): “This observation has inspired several researchers to try to improve the robustness and reliability of DNNs [9, 13, 17, 30, 32, 46, 51, 53, 62, 84, 89, 90].”
Safety Verification of Deep Neural Networks (2016, methodology): “We also mention that [38] uses several distortions such as JPEG compression, thumbnail resizing, random cropping, etc, to test the robustness of the trained network.”
Is Deep Learning Safe for Robot Vision? Adversarial Examples Against the iCub Humanoid (2017, background): “A different approach has been proposed in [31]: it aims at improving the stability of the deep feature space mapping by retraining the network using an objective function that penalizes examples (images) that are close in input space but lie far in deep feature space.”
Formal Guarantees on the Robustness of a Classifier against Adversarial Manipulation (2017, background): “While countermeasures have been proposed [8, 7, 26, 18, 12, 2], none of them provides a guarantee”
Theoretically Principled Trade-off between Robustness and Accuracy (2019, methodology): “A related line of research is adversarial training by regularization (Kurakin et al., 2017; Ross & Doshi-Velez, 2017; Zheng et al., 2016).”
Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey (2018, background): “[116] to improve the robustness of neural networks against small distortions to input images.”
Ground-Truth Adversarial Examples (2017, methodology): “Many effective techniques have been proposed for generating adversarial examples [2, 5, 16, 19]; and, conversely, several techniques have been proposed for training networks that are more robust to these examples [6–8, 15, 20].”
EAD: Elastic-Net Attacks to Deep Neural Networks via Adversarial Examples (2017, background): “Modifying the training loss or the network architecture to increase the robustness of DNNs to adversarial examples has been proposed in (Zheng et al. 2016; Madry et al. 2017; Tramèr et al. 2017; Zantedeschi, Nicolae, and Rawat 2017).”
Unlabeled Data Improves Adversarial Robustness (2019, methodology): “For the supervised robust loss, we use a robustness-promoting regularization term proposed in [56] and closely related to earlier proposals in [57, 30, 20].”
Are Labels Required for Improving Adversarial Robustness? (2019, neutral): “2, 3, 4, 5, 8, 9, 13, 14 [55] S.”