Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search

By Xiangxiang Chu, Tianbao Zhou, Bo Zhang, Jixiang Li. 2019

359 citations recorded by the source corpus.

Differentiable Architecture Search (DARTS) is now a widely disseminated weight-sharing neural architecture search method. However, it suffers from well-known performance collapse due to an inevitable aggregation of skip connections. In this paper, we first disclose that its root cause lies in an unfair advantage in exclusive competition. Through experiments, we show that if either of two conditions is broken, the collapse disappears. Thereby, we present a novel approach called Fair DARTS where the exclusive competition is relaxed to be collaborative. Specifically, we let each operation's architectural weight be independent of others. Yet there is still an important issue of discretization discrepancy. We then propose a zero-one loss to push architectural weights towards zero or one, which approximates an expected multi-hot solution. Our experiments are performed on two mainstream search spaces, and we derive new state-of-the-art results on CIFAR-10 and ImageNet. Our code is available on this https URL .

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Wei-Wei Tu
2. Quanming Yao
3. Yang Yu
4. Qiang Yang
5. Yi-Qi Hu
6. Xiangxiang Chu
7. Isabelle M Guyon
8. Bo Zhang

In Computer Vision and Pattern Recognition

1. Bo Zhang
2. Quanming Yao
3. Hugo Jair Escalante
4. Isabelle M Guyon
5. Xiangxiang Chu
6. Xiaolin Wei
7. Yang Yu
8. Qiang Yang

Citations in the corpus

• Automated Machine Learning: From Principles to Practices (2018, methodology): “Hyperparameters, such as “temperature” for controlling randomness and “max_token” for determining generation length, significantly influence the model’s output [143].”
• DARTS-: Robustly Stepping out of Performance Collapse Without Indicators (2020, methodology): “On the other hand, it also breaks the unfair advantage (Chu et al., 2020c) as the advantageous contribution from the residual block is factored out.”
• Auto-Panoptic: Cooperative Multi-Component Architecture Search for Panoptic Segmentation (2020, neutral): “Moreover, the shared weights across a wide range of architectures become ineffective and unpredictable in the ranking process since the architecture parameters and supernet weights are deeply coupled and performance collapse is often observed during optimization [9, 15, 8].”
• Sample-Efficient Neural Architecture Search by Learning Action Space (2019, neutral): “3 FairDARTS ([36]) c/o X 3.”