InstaNAS: Instance-aware Neural Architecture Search

By An-Chieh Cheng, Chieh Hubert Lin, Da-Cheng Juan, Wei Wei, Min Sun. 2018

54 citations recorded by the source corpus.

Conventional Neural Architecture Search (NAS) aims at finding a single architecture that achieves the best performance, which usually optimizes task related learning objectives such as accuracy. However, a single architecture may not be representative enough for the whole dataset with high diversity and variety. Intuitively, electing domain-expert architectures that are proficient in domain-specific features can further benefit architecture related objectives such as latency. In this paper, we propose InstaNAS—an instance-aware NAS framework—that employs a controller trained to search for a “distribution of architectures” instead of a single architecture; This allows the model to use sophisticated architectures for the difficult samples, which usually comes with large architecture related cost, and shallow architectures for those easy samples. During the inference phase, the controller assigns each of the unseen input samples with a domain expert architecture that can achieve high accuracy with customized inference costs. Experiments within a search space inspired by MobileNetV2 show InstaNAS can achieve up to 48.8% latency reduction without compromising accuracy on a series of datasets against MobileNetV2.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Yong Guo
2. Wuyang Chen
3. Jian Chen
4. Zhangyang Wang
5. Mingkui Tan
6. Xinyu Gong
7. S. Cong
8. Yin Zheng

Citations in the corpus

• AutoGAN-Distiller: Searching to Compress Generative Adversarial Networks (2020, neutral): “g, (Wu et al., 2019; Cheng et al., 2018; Tan & Le, 2019; Tan et al., 2019; Cheng et al., 2018), although most of them focused on compressing deep classifiers.”
• Retinex-inspired Unrolling with Cooperative Prior Architecture Search for Low-light Image Enhancement (2020, unknown): “Another group of approaches [3, 31] employ reinforcement learning techniques to train a meta-controller to generate potential architectures.”
• FasterSeg: Searching for Faster Real-time Semantic Segmentation (2019, methodology): “These works optimize both accuracy and resource utilization, via a combined loss function (Wu et al., 2019), or a hybrid reward signal for policy learning (Tan et al., 2019; Cheng et al., 2018), or a constrained optimization formulation (Dai et al., 2019).”
• A review of convolutional neural network architectures and their optimizations (2022, methodology): “2020a), reinforcement learning (Zoph and Le 2016; Tan et al. 2019; Zhong et al. 2020; Cheng et al. 2020), gradient descent-based method (Chen and Hsieh 2020; Chu et al.”
• Breaking the Curse of Space Explosion: Towards Efficient NAS with Curriculum Search (2020, methodology): “Cheng et al. (2018) propose an InstaNAS method that uses a dynamic reward function to gradually increase task difficulty, and Zoph & Le (2017) propose to gradually increase the number of layers.”
• Mitigating Forgetting in Online Continual Learning via Instance-Aware Parameterization (2020, methodology): “InstaNAS [7] leverages the one-shot architecture search consisting of a meta-graph and controller to select a path for each instance (i.e., instance-awareness) in order to improve the overall inference speed and accuracy.”