Approximate Fisher Information Matrix to Characterize the Training of Deep Neural Networks
By Zhibin Liao, Tom Drummond, Ian David Reid, Gustavo Carneiro. 2018
27 citations recorded by the source corpus.
In this paper, we introduce a novel methodology for characterizing the performance of deep learning networks (ResNets and DenseNet) with respect to training convergence and generalization as a function of mini-batch size and learning rate for image classification. This methodology is based on novel measurements derived from the eigenvalues of the approximate Fisher information matrix, which can be efficiently computed even for high capacity deep models. Our proposed measurements can help practitioners to monitor and control the training process (by actively tuning the mini-batch size and learning rate) to allow for good training convergence and generalization. Furthermore, the proposed measurements also allow us to show that it is possible to optimize the training process with a new dynamic sampling training approach that continuously and automatically change the mini-batch size and learning rate during the training process. Finally, we show that the proposed dynamic sampling training approach has a faster training time and a competitive classification accuracy compared to the current state of the art.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Matteo Boschini
- Simone Calderara
- Angelo Porrello
- Philipp Hennig
- Frederik Kunstner
- Pietro Buzzega
- Christa Zoufal
- Stefan Woerner
In Computer Vision and Pattern Recognition
- Matteo Boschini
- Simone Calderara
- Angelo Porrello
- Frederik Kunstner
- Philipp Hennig
- Lukas Balles
- Pietro Buzzega
- Davide Abati
Citations in the corpus
- Dark Experience for General Continual Learning: a Strong, Simple Baseline (2020, methodology): “According to (Liao et al. , 2018), this indicates that the training process leaves the optimal convergence/generalization region, thus empirically confirming the effectiveness of soft labels in continual learning scenarios.”
- Limitations of the Empirical Fisher Approximation (2019, methodology): “It has also been employed to explain certain characteristics of SGD [Zhu et al., 2018, Jastrzebski et al., 2018] or as a diagnostic tool during training [Liao et al., 2018].”
- The power of quantum neural networks (2020, methodology): “This behaviour has also been reported for the Hessian matrix, which coincides with the Fisher information matrix under certain conditions [33,51,52].”