Training Certifiably Robust Neural Networks with Efficient Local Lipschitz Bounds
By Yujia Huang, Huan Zhang, Yuanyuan Shi, J. Kolter, Anima Anandkumar. 2021
100 citations recorded by the source corpus.
Certified robustness is a desirable property for deep neural networks in safety-critical applications, and popular training algorithms can certify robustness of a neural network by computing a global bound on its Lipschitz constant. However, such a bound is often loose: it tends to over-regularize the neural network and degrade its natural accuracy. A tighter Lipschitz bound may provide a better tradeoff between natural and certified accuracy, but is generally hard to compute exactly due to non-convexity of the network. In this work, we propose an efficient and trainable \emph{local} Lipschitz upper bound by considering the interactions between activation functions (e.g. ReLU) and weight matrices. Specifically, when computing the induced norm of a weight matrix, we eliminate the corresponding rows and columns where the activation function is guaranteed to be a constant in the neighborhood of each given data point, which provides a provably tighter bound than the global Lipschitz constant of the neural network. Our method can be used as a plug-in module to tighten the Lipschitz bound in many certifiable training algorithms. Furthermore, we propose to clip activation functions (e.g., ReLU and MaxMin) with a learnable upper threshold and a sparsity loss to assist the network to achieve an even tighter local Lipschitz bound. Experimentally, we show that our method consistently outperforms state-of-the-art methods in both clean and certified accuracy on MNIST, CIFAR-10 and TinyImageNet datasets with various network architectures.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Matteo Boschini
- Lorenzo Bonicelli
- Simone Calderara
- Concetto Spampinato
- Angelo Porrello
- Bahram Zonooz
- Elahe Arani
- Gyuhak Kim
In Computer Vision and Pattern Recognition
- Matteo Boschini
- Simone Calderara
- Lorenzo Bonicelli
- Angelo Porrello
- Concetto Spampinato
- Hang Su
- Jun Zhu
- Liyuan Wang
Citations in the corpus
- On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learning (2022, positive): “Notably, it has been proved to provide a tighter upper bound (see Theorem 1 of [33]), and hence a more effective regularizing signal.”