Improving Neural Architecture Search Image Classifiers via Ensemble Learning
By Vladimir Macko, Charles Weill, Hanna Mazzawi, Javier Gonzalvo. 2019
18 citations recorded by the source corpus.
Finding the best neural network architecture requires significant time, resources, and human expertise. These challenges are partially addressed by neural architecture search (NAS) which is able to find the best convolutional layer or cell that is then used as a building block for the network. However, once a good building block is found, manual design is still required to assemble the final architecture as a combination of multiple blocks under a predefined parameter budget constraint. A common solution is to stack these blocks into a single tower and adjust the width and depth to fill the parameter budget. However, these single tower architectures may not be optimal. Instead, in this paper we present the AdaNAS algorithm, that uses ensemble techniques to compose a neural network as an ensemble of smaller networks automatically. Additionally, we introduce a novel technique based on knowledge distillation to iteratively train the smaller networks using the previous ensemble as a teacher. Our experiments demonstrate that ensembles of networks improve accuracy upon a single neural network while keeping the same number of parameters. Our models achieve comparable results with the state-of-the-art on CIFAR-10 and sets a new state-of-the-art on CIFAR-100.
Subfields: Biophysics, Computer Vision and Pattern Recognition, Media Technology
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Biophysics
In Computer Vision and Pattern Recognition
- Dacheng Tao
- Luc Van Gool
- Trevor Darrell
- Arber Zela
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
- Yi Yang
In Media Technology
Citations in the corpus
- Knowledge Distillation: A Survey (2020, methodology): “Of course, there are other special interesting applications of knowledge distillation, such as neural architecture search (Macko et al. 2019; Bashivan et al. 2019), interpretability of deepneural networks (Liu et al.”
- Neural Ensemble Search for Performant and Calibrated Predictions (2020, background): “[38] iteratively add (sub-)networks to an ensemble to improve the ensemble’s performance.”