Firefly Neural Architecture Descent: a General Approach for Growing Neural Networks

By Lemeng Wu, Peter Stone, Qiang Liu. 2021

74 citations recorded by the source corpus.

We propose firefly neural architecture descent, a general framework for progressively and dynamically growing neural networks to jointly optimize the networks' parameters and architectures. Our method works in a steepest descent fashion, which iteratively finds the best network within a functional neighborhood of the original network that includes a diverse set of candidate network structures. By using Taylor approximation, the optimal network structure in the neighborhood can be found with a greedy selection procedure. We show that firefly descent can flexibly grow networks both wider and deeper, and can be applied to learn accurate but resource-efficient neural architectures that avoid catastrophic forgetting in continual learning. Empirically, firefly descent achieves promising results on both neural architecture search and continual learning. In particular, on a challenging continual image classification task, it learns networks that are smaller in size but have higher average accuracy than those learned by the state-of-the-art methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. Jianfeng Gao
3. Lu Yuan
4. Qun Liu
5. Xin Jiang
6. Liu Liu
7. Lifeng Shang
8. Yajing Kong

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Lu Yuan
3. Jianfeng Gao
4. Luc Van Gool
5. Xiyang Dai
6. Dongdong Chen
7. Ming-Hsuan Yang
8. Jianwei Yang

Citations in the corpus

• Balancing Stability and Plasticity through Advanced Null Space in Continual Learning (2022, methodology): “Architecture-based methods allocate different parameters or add new parameters for the new task, while sharing parameters across tasks to reduce the interference of previous tasks [3, 22, 27–29, 41, 42, 44, 53, 56, 60].”
• Residual Mixture of Experts (2022, methodology): “Inspired by Firefly Splitting [55], in RMoE, we select the layer that can maximally decrease the loss function in a layer-wise fashion.”
• LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning (2023, background): “The dynamic architecture approach gradually expands the learning model to incorporate new knowledge [56, 72, 41, 26, 71, 5].”
• Continual Learning and Private Unlearning (2022, background): “…this end, typical MU approaches consider training multiple models on different shards of data so that unlearning only requires retraining a specific model on part of the dataset (Bourtoule et al., 2019), or storing learned model parameters and their gradients for rapid retraining (Wu et al., 2020b).”
• bert2BERT: Towards Reusable Pretrained Language Models (2021, methodology): “To handle this problem, some works (Wu et al., 2019, 2020b; Wang et al., 2019b; Wu et al., 2020a) leverage a functional steepest descent idea to decide the optimal subset of neurons to be split.”