AutoGrow: Automatic Layer Growing in Deep Convolutional Networks

By Wujie Wen, Feng Yan, Hai Helen Li. 2019

41 citations recorded by the source corpus.

Depth is a key component of Deep Neural Networks (DNNs), however, designing depth is heuristic and requires many human efforts. We proposeAutoGrow to automate depth discovery in DNNs: starting from a shallow seed architecture,AutoGrow grows new layers if the growth improves the accuracy; otherwise, stops growing and thus discovers the depth. We propose robust growing and stopping policies to generalize to different network architectures and datasets. Our experiments show that by applying the same policy to different network architectures,AutoGrow can always discover near-optimal depth on various datasets of MNIST, FashionMNIST, SVHN, CIFAR10, CIFAR100 and ImageNet. For example, in terms of accuracy-computation trade-off,AutoGrow discovers a better depth combination in \resnets than human experts. OurAutoGrow is efficient. It discovers depth within similar time of training a single DNN. Our code is available at \urlhttps://github.com/wenwei202/autogrow.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Industrial and Manufacturing Engineering, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Luke S. Zettlemoyer
2. Sergey Levine
3. Quoc V. Le
4. Jianfeng Gao
5. Yejin Choi
6. Percy Liang
7. Mohit Bansal
8. Pieter Abbeel

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Ming-Hsuan Yang
4. Trevor Darrell
5. Jitendra Malik
6. Xiaogang Wang
7. Andrew Zisserman
8. Andrea Vedaldi

In Industrial and Manufacturing Engineering

1. Peter Stone
2. Qiang Liu
3. Lemeng Wu

In Model Architectures

1. Alec Radford
2. Rewon Child
3. Peter Stone
4. Jiawei Han
5. Scott Gray
6. Sen Lin
7. Chen Chen
8. Jing Li

Citations in the corpus

• Scaling Laws for Neural Language Models (2020, methodology): “And using methods like [WRH17, WYL19], which grow networks as they train, it might be possible to remain on the compute-efficient frontier for an entire training run.”
• Firefly Neural Architecture Descent: a General Approach for Growing Neural Networks (2021, methodology): “For AutoGrow, we use the result report in the original paper.”
• On the Transformer Growth for Progressive BERT Training (2020, background): “AutoGrow (Wen et al., 2020) attempts to automate the discover of proper depth to achieve near-optimal performance on different datasets.”
• GROWN: GRow Only When Necessary for Continual Learning (2021, methodology): “Recently proposed Autogrow (Wen et al. 2020) takes an AutoML approach to grow layers.”