Large Batch Training of Convolutional Networks with Layer-wise Adaptive Rate Scaling

44 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Bag of Tricks for Image Classification with Convolutional Neural Networks (2018, methodology): “endation, it only applies the weight decay to the weights in convolution and fullyconnected layers. Other parameters, including the biases and and in BN layers, are left unregularized. Note that LARS [4] offers layer-wise adaptive learning rate and is reported to be effective for extremely large bat”
On the Difficulty of Defending Self-Supervised Learning against Model Extraction (2022, methodology): “In the case of stealing from the ImageNet victim model, we use a larger learning rate of 0.1 or 1.0 with the LARS optimizer (Ginsburg et al., 2018) and a batch size of 256 or 512.”
A Hitchhiker's Guide On Distributed Training of Deep Neural Networks (2018, methodology): “arning rate. However, training with large batch sizes is a promising avenue as it can speed up training time dramatically by lowering the training time from days to minutes as evidenced in [9], [14], [13] and the work by [11] reinforces this trend by empirically proving that increasing batch size is”