Hessian-based Analysis of Large Batch Training and Robustness to Adversaries

By Zi-Mei Yao, A. Gholami, Qi Lei, Kurt Keutzer, Michael W. Mahoney. 2018

194 citations recorded by the source corpus.

Large batch size training of Neural Networks has been shown to incur accuracy loss when trained with the current methods. The exact underlying reasons for this are still not completely understood. Here, we study large batch size training through the lens of the Hessian operator and robust optimization. In particular, we perform a Hessian based study to analyze exactly how the landscape of the loss function changes when training with large batch size. We compute the true Hessian spectrum, without approximation, by back-propagating the second derivative. Extensive experiments on multiple networks show that saddle-points are not the cause for generalization gap of large batch size training, and the results consistently show that large batch converges to points with noticeably higher Hessian spectrum. Furthermore, we show that robust training allows one to favor flat areas, as points with large Hessian spectrum show poor robustness to adversarial perturbation. We further study this relationship, and provide empirical and theoretical proof that the inner loop for robust training is a saddle-free optimization problem \textit{almost everywhere}. We present detailed experiments with five different network architectures, including a residual network, tested on MNIST, CIFAR-10, and CIFAR-100 datasets. We have open sourced our method which can be accessed at [1].

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

An Investigation into Neural Net Optimization via Hessian Eigenvalue Density (2019, background): “Previous studies on the Hessian have focused on small models, or are limited to computing only a few eigenvalues [23, 24, 30].”
Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT (2019, methodology): “However, the Hessian spectrum can be computed by a matrix-free power iteration method [39], and this does not require explicit formation of the operator.”
Don't Use Large Mini-Batches, Use Local SGD (2018, methodology): “Even if the sharp minima have the same training loss as the flat ones, small discrepancies of the data can easily lead to poor generalization performance (Hochreiter & Schmidhuber, 1997; Keskar et al., 2016; Yao et al., 2018).”
Understanding and Robustifying Differentiable Architecture Search (2019, background): “Recent work [Keskar et al., 2016, Yao et al., 2018] focuses more on the settings of large/small batch size training, where observations show that small batch training tends to get attracted to ﬂatter minima and generalizes better.”
On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learning (2022, background): “In this respect, the bridge between the two objectives is worth-exploring and still open to debate [83, 71, 87, 36].”
HAWQ: Hessian AWare Quantization of Neural Networks With Mixed-Precision (2019, methodology): “However, it is possible to compute the Hessian eigenvalues without explicitly forming it, using a matrix-free power iteration algorithm [39], [23], [38].”
On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent (2018, background): “However, there is a growing body of theoretical and empirical results suggesting that large batch sizes adversely affect the generalization performance of the ﬁnal model [31, 17, 7].”
The Implicit and Explicit Regularization Effects of Dropout (2020, background): “Keskar et al. (2016); Yao et al. (2018); Jastrzebski et al. (2018) study the relationship between SGD batch size and notions of “ﬂatness” of local minima via metrics related to the magnitudes of the eigenvalues of the second derivative of the loss with respect to the model parameters.”
Emergent properties of the local geometry of neural loss landscapes (2019, background): “Recent works have observed this phenomenon in small networks [7, 8, 9], as well as large networks [10, 11].”
Characterizing possible failure modes in physics-informed neural networks (2021, background): “This tends to be more informative than perturbing the model parameters in random directions [41, 42].”
ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning (2020, methodology): “However, one could use Randomized Numerical Linear Algebra to get a sketch of the Hessian matrix [23, 66, 67].”
Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning (2018, methodology): “This includes: work on large-batch learning and the generalization gap [148, 72, 64, 57, 67, 131, 66, 146, 94, 151, 152]; work on Energy Landscape approaches to NN training [70, 149, 44, 123, 30, 29, 27, 65, 47, 12, 104, 150, 50, 83, 82, 95]; work on using weight matrices or properties of weight mat”
A Diffusion Theory for Deep Learning Dynamics: Stochastic Gradient Descent Escapes From Sharp Minima Exponentially Fast (2020, background): “They try to measure ﬂatness (Hochreiter and Schmidhuber, 1997; Keskar et al., 2017; Sagun et al., 2017; Yao et al., 2018), rescale ﬂatness (Tsuzuku et al., 2019), and ﬁnd ﬂatter minima (Hoffer et al., 2017; Chaudhari et al., 2017; He et al., 2019b).”
Understanding Impacts of High-Order Loss Approximations and Features in Deep Learning Interpretation (2019, background): “Yao et al. 2018 (Yao et al., 2018) also proved the Hessian for piecewise linear networks is at most rank c but did not derive the exact input Hessian.”
Logit Pairing Methods Can Fool Gradient-Based Attacks (2018, background): “Many recent papers [Buckman et al., 2018, Kannan et al., 2018, Yao et al., 2018] that claim robustness of their models mainly rely on the PGD attack from [Madry et al., 2017] with the default settings.”
PyHessian: Neural Networks Through the Lens of the Hessian (2019, methodology): “PYHESSIAN computes Hessian information by applying known techniques from Numerical Linear Algebra (NLA) [4, 10, 17] and Randomized NLA (RandNLA) [3, 7, 8, 18, 25, 27] (that are approximate but come with rigorous theory).”
On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length (2018, background): “Finally, Yao et al. (2018) study the relation between the Hessian and adversarial robustness at the endpoint of training.”