Evolutionary-Neural Hybrid Agents for Architecture Search

By Krzysztof Maziarz, Andrey Khorlin, Quentin de Laroussilhe, Andrea Gesmundo. 2018

44 citations recorded by the source corpus.

Neural Architecture Search has shown potential to automate the design of neural networks. Deep Reinforcement Learning based agents can learn complex architectural patterns, as well as explore a vast and compositional search space. On the other hand, evolutionary algorithms offer higher sample efficiency, which is critical for such a resource intensive application. In order to capture the best of both worlds, we propose a class of Evolutionary-Neural hybrid agents (Evo-NAS). We show that the Evo-NAS agent outperforms both neural and evolutionary agents when applied to architecture search for a suite of text and image classification benchmarks. On a high-complexity architecture search space for image classification, the Evo-NAS agent surpasses the accuracy achieved by commonly used agents with only 1/3 of the search cost.

Subfields: Artificial Intelligence, Neural Architecture Search & AutoML, Evolutionary Computation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Mengjie Zhang
2. Bing Xue
3. Yanan Sun
4. Gary G. Yen
5. Colin White
6. Yash Savani
7. Sam Nolen
8. Yuqiao Liu

In Neural Architecture Search & AutoML

1. Sam Nolen
2. Yash Savani
3. Colin White
4. Willie Neiswanger
5. Bing Xue
6. Mengjie Zhang
7. Gary G. Yen
8. Yuqiao Liu

In Evolutionary Computation

1. Yash Savani
2. Colin White
3. Willie Neiswanger
4. Sam Nolen
5. Bing Xue
6. Mengjie Zhang
7. Gary G. Yen
8. Yuqiao Liu

Citations in the corpus

• BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search (2019, methodology): “Some of the most popular recent techniques for NAS include evolutionary algorithms (Maziarz et al. 2018), reinforcement learning (Zoph and Le 2017; Pham et al.”
• A Survey on Evolutionary Neural Architecture Search (2020, methodology): “Unlike CIFAR-10 and CIFAR-100, which are commonly used in ENAS, fewer methods choose to verify their performance on ImageNet [122], [134], [180].”
• muNet: Evolving Pretrained Deep Neural Networks into Scalable Auto-tuning Multitask Systems (2022, background): “Sample efficient evolutionary approaches have been also proposed [Real et al., 2019, Maziarz et al., 2018], Furthermore, more efficient parameter-sharing approaches have been proposed [Pham et al., 2018, Liu et al., 2019a, Kokiopoulou et al., 2019] that connect the NAS field with the one of routing…”
• A Study on Encodings for Neural Architecture Search (2020, methodology): “Some of the most popular techniques for NAS include evolutionary algorithms [11], reinforcement learning [12, 19], Bayesian optimization [6], gradient descent [10], neural predictors [21], and local search [23].”
• Local Search is State of the Art for NAS Benchmarks (2020, background): “Neural architecture search (NAS) has gained significant attention in recent years (Kitano, 1990; Stanley and Miikkulainen, 2002; Zoph and Le, 2017), although the first few techniques have been around since at least the 1990s (Shah et al., 2018; Maziarz et al., 2018).”
• Local Search is State of the Art for Neural Architecture Search Benchmarks (2020, background): “NAS has gained significant attention in recent years [40], although the first few techniques have been around since at least the 1990s [22, 29].”