Transfer Learning with Neural AutoML

By Catherine Wong, Neil Houlsby, Yifeng Lu, Andrea Gesmundo. 2018

118 citations recorded by the source corpus.

We reduce the computational cost of Neural AutoML with transfer learning. AutoML relieves human effort by automating the design of ML algorithms. Neural AutoML has become popular for the design of deep learning architectures, however, this method has a high computation cost. To address this we propose Transfer Neural AutoML that uses knowledge from prior tasks to speed up network design. We extend RL-based architecture search methods to support parallel training on multiple tasks and then transfer the search strategy to new tasks. On language and image classification data, Transfer Neural AutoML reduces convergence time over single-task training by over an order of magnitude on many tasks.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

Parameter-Efficient Transfer Learning for NLP (2019, methodology): “Specifically, we run the single-task Neural AutoML algorithm, similar to Zoph & Le (2017); Wong et al. (2018).”
AutoML: A Survey of the State-of-the-Art (2019, methodology): “[131] propose Transfer Neural AutoML, which uses knowledge from prior tasks to accelerate the process of network design.”
A Survey on Neural Architecture Search (2019, background): “Xie and Yuille (2017) consider a similar search space which uses summation instead of concatenation as the merging operation.”
EAT-NAS: elastic architecture transfer for accelerating large-scale neural architecture search (2019, methodology): “There is a work [25] that combines transfer learning with RL-based NAS method.”
Taking Human out of Learning Applications: A Survey on Automated Machine Learning (2018, background): “ﬁguration generation. In [112], Bayesian optimization is accompanied with multi-task Gaussian process models so that knowledge gained from past tuning tasks can be transfered to warm-start search. In [174], a multi-task neural AutoML controller is trained to learn hyper-parameters for neural network”
Fast Task-Aware Architecture Inference (2019, methodology): “• Methods based on Reinforcement Learning (RL) [22, 20, 15, 12] evaluate candidate child model architectures on-the-ﬂy, by training and evaluating on a validation set.”
InstaNAS: Instance-aware Neural Architecture Search (2018, background): “This is also similar in the spirit of [31], which accelerates NAS by transferring the previously learned policy to new tasks (the new reward bound in the case of InstaNAS ).”
Design Space for Graph Neural Networks (2020, background): “Meta-level architecture design has also been studied in [23, 27, 36, 49], with the assumption that different tasks follow the same distribution ( e.g. , variants of ImageNet dataset [2]).”
The Visual Task Adaptation Benchmark (2019, background): “For example: (i) Zoph & Le (2017) learn performant architectures, (ii) Bello et al. (2017) learn optimization algorithms, (iii) Wong et al. (2018) learn hyperparameter search strategies, (iv) Cubuk et al. (2019) learn data augmentation functions, (v) Raghu et al. (2019) learn layer-wise…”