FreezeOut: Accelerate Training by Progressively Freezing Layers

By Andrew Brock, Theodore Lim, James M. Ritchie, Nick Weston. 2017

159 citations recorded by the source corpus.

The early layers of a deep neural net have the fewest parameters, but take up the most computation. In this extended abstract, we propose to only train the hidden layers for a set portion of the training run, freezing them out one-by-one and excluding them from the backward pass. Through experiments on CIFAR, we empirically demonstrate that FreezeOut yields savings of up to 20% wall-clock time during training with 3% loss in accuracy for DenseNets, a 20% speedup without loss of accuracy for ResNets, and no improvement for VGG networks. Our code is publicly available at this https URL

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Eugene Belilovsky
2. Michael Eickenberg
3. Massimo Caccia
4. Simone Scardapane
5. Edouard Oyallon
6. Aurelio Uncini
7. Michele Scarpiniti
8. Enzo Baccarelli

In Computer Vision and Pattern Recognition

1. Eugene Belilovsky
2. Michael Eickenberg
3. Massimo Caccia
4. Edouard Oyallon
5. Simone Scardapane
6. Aurelio Uncini
7. Michal Drozdzal
8. Michele Scarpiniti

In Efficiency & Compression

1. Graham W. Taylor
2. Aurelio Uncini
3. Massimo Caccia
4. Simone Scardapane
5. Eugene Belilovsky
6. Michael Eickenberg
7. Edouard Oyallon

In Optimization & Training

1. Eugene Belilovsky
2. Massimo Caccia
3. Simone Scardapane
4. Aurelio Uncini
5. Michele Scarpiniti
6. Enzo Baccarelli
7. Michael Eickenberg
8. Edouard Oyallon

Citations in the corpus

• Why Should We Add Early Exits to Neural Networks? (2020, background): “For example, freezeout [13] is a variation of the JT approach, in which layers are iteratively ‘freezed’ by annealing their learning”
• Parameter Prediction for Unseen Deep Architectures (2021, background): “[5] Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston.”
• Shallow Learning For Deep Networks (2018, background): “Brock et al. (2017) builds a network in stages, where certain layers are progressively frozen, which permits faster training.”
• Learning where to learn: Gradient sparsity in meta and continual learning (2021, background): “Interestingly, similar effects have been reported in classic neural network training where both freezing layers throughout training (Raghu et al., 2017; Brock et al., 2017) and the use of large learning rates (Lewkowycz et al., 2020) seem to improve generalization.”
• Greedy Layerwise Learning Can Scale to ImageNet (2018, background): “Brock et al. (2017) builds a network in stages, where certain layers are progressively frozen, permitting faster training.”
• Dropout vs. batch normalization: an empirical study of their impact to deep learning (2020, unknown): “As a result, many methods/approaches, such as layer-wise freeze training [2], dropout [28], and batch normalization [7], have been proposed to ensure steady and efficient training of reliable deep learning models.”