Optimal Transport Kernels for Sequential and Parallel Neural Architecture Search
By Vu Nguyen, Tam Le, Makoto Yamada, Michael A. Osborne. 2020
43 citations recorded by the source corpus.
Neural architecture search (NAS) automates the design of deep neural networks. One of the main challenges in searching complex and non-continuous architectures is to compare the similarity of networks that the conventional Euclidean metric may fail to capture. Optimal transport (OT) is resilient to such complex structure by considering the minimal cost for transporting a network into another. However, the OT is generally not negative definite which may limit its ability to build the positive-definite kernels required in many kernel-dependent frameworks. Building upon tree-Wasserstein (TW), which is a negative definite variant of OT, we develop a novel discrepancy for neural architectures, and demonstrate it within a Gaussian process surrogate model for the sequential NAS settings. Furthermore, we derive a novel parallel NAS, using quality k-determinantal point process on the GP posterior, to select diverse and high-performing architectures from a discrete set of candidates. Empirically, we demonstrate that our TW-based approaches outperform other baselines in both sequential and parallel NAS.
Subfields: Artificial Intelligence, Management Science and Operations Research
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yaochu Jin
- Colin White
- Yash Savani
- Markus Olhofer
- Willie Neiswanger
- Sebastian Schmitt
- Xilu Wang
- Handing Wang
In Management Science and Operations Research
- Yaochu Jin
- Markus Olhofer
- Yash Savani
- Sebastian Schmitt
- Xilu Wang
- Colin White
- Willie Neiswanger
- Ye Tian
Citations in the corpus
- BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search (2019, background): “Since its release, several papers have included BANANAS in new experiments, further showing that BANANAS is a competitive NAS algorithm (Krishna et al. 2020; Siems et al. 2020; Nguyen et al. 2020; Ru et al. 2020; Wei et al. 2020).”
- Recent Advances in Bayesian Optimization (2022, methodology): “be inefficient in many applications where multiple data points can be sampled in parallel [144].”