Neural Rejuvenation: Improving Deep Network Training by Enhancing Computational Resource Utilization
By Siyuan Qiao, Zhe L. Lin, Jianming Zhang, Alan L. Yuille. 2018
24 citations recorded by the source corpus.
In this paper, we study the problem of improving computational resource utilization of neural networks. Deep neural networks are usually over-parameterized for their tasks in order to achieve good performances, thus are likely to have underutilized computational resources. This observation motivates a lot of research topics, e.g. network pruning, architecture search, etc. As models with higher computational costs (e.g. more parameters or more computations) usually have better performances, we study the problem of improving the resource utilization of neural networks so that their potentials can be further realized. To this end, we propose a novel optimization method named Neural Rejuvenation. As its name suggests, our method detects dead neurons and computes resource utilization in real time, rejuvenates dead neurons by resource reallocation and reinitialization, and trains them with new training schemes. By simply replacing standard optimizers with Neural Rejuvenation, we are able to improve the performances of neural networks by a very large margin while using similar training efforts and maintaining their original resource usages. The code is available here: https://github.com/joe-siyuan-qiao/NeuralRejuvenation-CVPR19
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Christoph Feichtenhofer
- Greg Mori
- Kaiming He
- Philipp Krahenbuhl
- Ross B. Girshick
- Andrew Zisserman
- Luc Van Gool
- Jitendra Malik
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Greg Mori
- Jitendra Malik
- Christoph Feichtenhofer
- Andrew Zisserman
- Xiaogang Wang
- Kaiming He
- Trevor Darrell
Citations in the corpus
- Piggyback GAN: Efficient Lifelong Learning for Image Conditioned Generation (2021, background): “Parameter efficiency can also be encouraged during training via architecture design [23,29], sparsity inducing priors [33], resource re-allocation [26], or knowledge distillation from a teacher network [34,41].”
- A Multigrid Method for Efficiently Training Video Models (2019, background): “, [8, 24, 33, 39]), pre-training [3, 11], distributed training [13, 50], or advances in hardware [22] and software design [4, 6].”