iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradients

By Miao Zhang, Steven W. Su, Shirui Pan, Xiaojun Chang, Ehsan Abbasnejad, Reza Haffari. 2021

85 citations recorded by the source corpus.

\textit{Differentiable ARchiTecture Search} (DARTS) has recently become the mainstream of neural architecture search (NAS) due to its efficiency and simplicity. With a gradient-based bi-level optimization, DARTS alternately optimizes the inner model weights and the outer architecture parameter in a weight-sharing supernet. A key challenge to the scalability and quality of the learned architectures is the need for differentiating through the inner-loop optimisation. While much has been discussed about several potentially fatal factors in DARTS, the architecture gradient, a.k.a. hypergradient, has received less attention. In this paper, we tackle the hypergradient computation in DARTS based on the implicit function theorem, making it only depends on the obtained solution to the inner-loop optimization and agnostic to the optimization path. To further reduce the computational requirements, we formulate a stochastic hypergradient approximation for differentiable NAS, and theoretically show that the architecture optimization with the proposed method, named iDARTS, is expected to converge to a stationary point. Comprehensive experiments on two NAS benchmark search spaces and the common NAS search space verify the effectiveness of our proposed method. It leads to architectures outperforming, with large margins, those learned by the baseline methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Multiset-Equivariant Set Prediction with Approximate Implicit Differentiation (2021, methodology): “…a VJP to compute u (cid:62) B Previous uses of implicit differentiation in meta-learning (Rajeswaran et al., 2019) and neural architecture search (Zhang et al., 2021b) also involve solving a linear system, but their Y corresponds to the neural network parameters and thus usually has millions of…”
A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations (2023, unknown): “including neural network pruning ([19], [20]), low-rank factor-izations of the weight matrices ([21], [22]), quantization ([23], [24]), knowledge distillation ([25], [26]), neural architecture search ([27], [28]), and other techniques ([29], [30]).”