Teacher Guided Architecture Search

By Pouya Bashivan, Mark Tensen, James J. DiCarlo. 2018

29 citations recorded by the source corpus.

Much of the recent improvement in neural networks for computer vision has resulted from discovery of new networks architectures. Most prior work has used the performance of candidate models following limited training to automatically guide the search in a feasible way. Could further gains in computational efficiency be achieved by guiding the search via measurements of a high performing network with unknown detailed architecture (e.g. the primate visual system)? As one step toward this goal, we use representational similarity analysis to evaluate the similarity of internal activations of candidate networks with those of a (fixed, high performing) teacher network. We show that adopting this evaluation metric could produce up to an order of magnitude in search efficiency over performance-guided methods. Our approach finds a convolutional cell structure with similar performance as was previously found using other methods but at a total computational cost that is two orders of magnitude lower than Neural Architecture Search (NAS) and more than four times lower than progressive neural architecture search (PNAS). We further show that measurements from only ∼300 neurons from primate visual system provides enough signal to find a network with an Imagenet top-1 error that is significantly lower than that achieved by performance-guided architecture search alone. These results suggest that representational matching can be used to accelerate network architecture search in cases where one has access to some or all of the internal representations of a teacher network of interest, such as the brain’s sensory processing networks.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Electrical and Electronic Engineering

Citations in the corpus

A Comprehensive Survey of Neural Architecture Search (2020, background): “Therefore, a lot of work has emerged on how to reduce the amount of calculation and accelerate the search of the network architecture [18–20, 48, 49, 52, 84, 105].”
Knowledge Distillation: A Survey (2020, methodology): “…knowledge distillation is employed to improve the efﬁciency of neural architecture search, such as AdaNAS (Macko et al. 2019), NAS with distilled architecture knowledge (Li et al. 2020a), teacher guided search for architectures or TGSA (Bashivan et al. 2019), and one-shot NAS (Peng et al. 2020).”
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, background): “t years, NAS has become a popular topic in deep learning. NAS has the potential of automating the design of neural networks. The potential of NAS for GAN compression has been recently demonstrated in [21], [137]. It is shown to be effective for ﬁnding efﬁcient student architecture from the teacher m”