Evolutionary architecture search for deep multitask networks

By Jason Zhi Liang, Elliot Meyerson, Risto Miikkulainen. 2018

128 citations recorded by the source corpus.

Multitask learning, i.e. learning several tasks at once with the same neural network, can improve performance in each of the tasks. Designing deep neural network architectures for multitask learning is a challenge: There are many ways to tie the tasks together, and the design choices matter. The size and complexity of this problem exceeds human design ability, making it a compelling domain for evolutionary optimization. Using the existing state of the art soft ordering architecture as the starting point, methods for evolving the modules of this architecture and for evolving the overall topology or routing between modules are evaluated in this paper. A synergetic approach of evolving custom routings with evolved, shared modules for each task is found to be very powerful, significantly improving the state of the art in the Omniglot multitask, multialphabet character recognition domain. This result demonstrates how evolution can be instrumental in advancing deep neural network and complex system design in general.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Bayesian Neural Architecture Search using A Training-Free Performance Metric (2020, methodology): “For RNNs, NEAT-LSTM [39] and CoDeepNeat [30] extend NEAT to mitigate its limitations when evolving the topology and weights of the network.”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “More recent work (Devin et al., 2017; Frans et al., 2017; Fernando et al., 2017; Rosenbaum et al., 2017; Meyerson and Miikkulainen, 2017; Ramachandran and Le, 2018; Chang et al., 2018; Liang et al., 2018; Alet et al., 2018; Cases et al., 2019; Yang et al., 2020) has focused on training neural networ”
Modular meta-learning (2018, methodology): “[22] expanded these ideas to the multitask setting with two particular compositional schemes: soft layer ordering and routing in DAGs.”
NSGA-NET: A Multi-Objective Genetic Algorithm for Neural Architecture Search (2018, background): “CoDeepNEAT does, however, produce state-of-the-art results in the Omniglot multi-task learning domain Liang et al. (2018).”
Latent Multi-Task Architecture Learning (2017, methodology): “Liu, Qiu, and Huang (2017) use a similar technique for sentiment analysis.”
Multi-Task Learning with Deep Neural Networks: A Survey (2020, methodology): “On the other hand, (Liang et al., 2018) introduces several variations of a multi-task neural architecture search algorithm that uses evolutionary strategies to learn neural network modules which can be reordered differently for various tasks.”
Random error sampling-based recurrent neural network architecture optimization (2019, methodology): “Focusing on RNNs, NEAT-LSTM [33] and CoDeepNeat [34] extend NEAT to mitigate its limitations when evolving the topologies and weights of the network.”
MTL-NAS: Task-Agnostic Neural Architecture Search Towards General-Purpose Multi-Task Learning (2020, methodology): “More recently, MTL and multi-modal learning have been exploited using NAS [32, 45].”
Multi-Task Learning for Dense Prediction Tasks: A Survey (2020, methodology): “[97] implemented an evolutionary architecture search for MTL, while others explored alternatives like branched MTL networks [9], [10], [11], [12], routing [65], stochastic filter grouping [56], and feature partitioning [68].”
A Specialized Evolutionary Strategy Using Mean Absolute Error Random Sampling to Design Recurrent Neural Networks (2019, methodology): “Focusing on RNNs, NEAT-LSTM [34] and CoDeepNeat [35] extend NEAT to mitigate its limitations when evolving the topologies and weights of the network.”
A Survey on Evolutionary Neural Architecture Search (2020, methodology): “The problem is: can a method search the corresponding type of neural network for multiple tasks in a large encoding space, including all the popular widely used components? Instead of searching one multitask network [212] which learns several tasks at once with the same neural network, the aim is to”
NSGA-Net: neural architecture search using multi-objective genetic algorithm (2018, background): “CoDeepNEAT does, however, produce state-of-the-art results in the Omniglot multi-task learning domain [24].”
Deeper Insights into Weight Sharing in Neural Architecture Search (2020, background): “Population based algorithms (Gaier & Ha, 2019; Liang et al., 2018; Jaderberg et al., 2017) are another popular approach for NAS, new trials could inherit neural architecture from better performing ones as well as their weights, and mutate the architecture to explore better ones.”
Branched Multi-Task Networks: Deciding what layers to share (2019, methodology): “To alleviate the heavy computation burden, a recent work [23] implemented an evolutionary architecture search for multi-task networks, while other researchers explored more viable alternatives, like routing [39], stochastic filter grouping [5], and feature partitioning [35].”
Gumbel-Matrix Routing for Flexible Multi-task Learning (2019, methodology): “We follow previous works (Liang et al., 2018; Ramachandran & Le, 2019) and use a fixed random subset of 20 alphabets, splitting every alphabet into training/validation/test sets with proportions 50%/20%/30%.”
Evolutionary neural AutoML for deep learning (2019, background): “LEAF extends this MTL architecture by coevolving both the module architectures and the blueprint (routing between the modules) of the DNN [30].”