Fast Task-Aware Architecture Inference

By Efi Kokiopoulou, Anja Hauth, Luciano Sbaiz, Andrea Gesmundo, Gábor Bartók, Jesse Berent. 2019

18 citations recorded by the source corpus.

Neural architecture search has been shown to hold great promise towards the automation of deep learning. However in spite of its potential, neural architecture search remains quite costly. To this point, we propose a novel gradient-based framework for efficient architecture search by sharing information across several tasks. We start by training many model architectures on several related (training) tasks. When a new unseen task is presented, the framework performs architecture inference in order to quickly identify a good candidate architecture, before any model is trained on the new task. At the core of our framework lies a deep value network that can predict the performance of input architectures on a task by utilizing task meta-features and the previous model training experiments performed on related tasks. We adopt a continuous parametrization of the model architecture which allows for efficient gradient-based optimization. Given a new task, an effective architecture is quickly identified by maximizing the estimated performance with respect to the model architecture parameters with simple gradient ascent. It is key to point out that our goal is to achieve reasonable performance at the lowest cost. We provide experimental results showing the effectiveness of the framework despite its high computational efficiency.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

A Survey on Neural Architecture Search (2019, methodology): “Kokiopoulou et al. (2019) Kokiopoulou et al. (2019) propose to automate fine-tuning for transferring models across different text classification tasks by using a surrogate model.”
muNet: Evolving Pretrained Deep Neural Networks into Scalable Auto-tuning Multitask Systems (2022, background): “…proposed [Real et al., 2019, Maziarz et al., 2018], Furthermore, more efﬁcient parameter-sharing approaches have been proposed [Pham et al., 2018, Liu et al., 2019a, Kokiopoulou et al., 2019] that connect the NAS ﬁeld with the one of routing networks [Fernando et al., 2017, Maziarz et al., 2019].”