NAT: Neural Architecture Transformer for Accurate and Compact Architectures

By Yong Guo, Yin Zheng, Mingkui Tan, Qi Chen, Jian Chen, Peilin Zhao, Junzhou Huang. 2019

101 citations recorded by the source corpus.

Designing effective architectures is one of the key factors behind the success of deep neural networks. Existing deep architectures are either manually designed or automatically searched by some Neural Architecture Search (NAS) methods. However, even a well-searched architecture may still contain many non-significant or redundant modules or operations (e.g., convolution or pooling), which may not only incur substantial memory consumption and computation cost but also deteriorate the performance. Thus, it is necessary to optimize the operations inside an architecture to improve the performance without introducing extra computation cost. Unfortunately, such a constrained optimization problem is NP-hard. To make the problem feasible, we cast the optimization problem into a Markov decision process (MDP) and seek to learn a Neural Architecture Transformer (NAT) to replace the redundant operations with the more computationally efficient ones (e.g., skip connection or directly removing the connection). Based on MDP, we learn NAT by exploiting reinforcement learning to obtain the optimization policies w.r.t. different architectures. To verify the effectiveness of the proposed strategies, we apply NAT on both hand-crafted architectures and NAS based architectures. Extensive experiments on two benchmark datasets, i.e., CIFAR-10 and ImageNet, demonstrate that the transformed architecture by NAT significantly outperforms both its original form and those architectures optimized by existing methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Hierarchical Neural Architecture Search for Single Image Super-Resolution (2020, methodology): “Speciﬁcally, NAS methods seek to ﬁnd the optimal cell architecture [14]–[16] or a whole network architecture [17].”
Understanding and Overcoming the Challenges of Efficient Transformer Quantization (2021, methodology): “…tasks (Houlsby et al., 2019; Stickland and Murray, 2019), knowledge distillation (Sanh et al., 2019; Jiao et al., 2020), neural architecture search (Guo et al., 2019; Wang et al., 2020a), pruning (Sanh et al., 2020; Prasanna et al., 2020), and better pre-training (Liu et al., 2019; Clark et al.,…”
A Survey on Vision Transformer (2020, background): “mated with neural architecture search (NAS) [80], [200], which automatically searches how to combine different components.”
Discrimination-Aware Network Pruning for Deep Model Compression (2020, background): “ith the Computer Science and Engineering department, The University of Texas at Arlington. E-mail: jzhuang@uta.edu image generation [5 ], [13 15], video analysis [63 76 83] and many other areas [14], [20]. However, the large model size and high computation cost remain great obstacles for many applic”
Location-Aware Graph Convolutional Networks for Video Question Answering (2020, background): “Recently, deep learning has witnessed a great process (Tan, Tsang, and Wang 2014; Cao et al. 2018; 2019; Gan et al. 2019; Guo et al. 2019a).”
Efficient Transformers: A Survey (2020, methodology): “These models incorporate the notion of a global model memory, which is reminiscent of the Set Transformer’s inducing point method or the global model memory of the Star Transformer.”
Towards Accurate Text-based Image Captioning with Content Diversity Exploration (2021, methodology): “RL-based methods [18, 30, 35, 39] model the sequence generation as Markov Decision Process [47] and directly maximise the metric scores.”
Dense Regression Network for Video Grounding (2020, background): “Recently, great progress has been achieved in deep learning [48, 47, 2, 15, 13, 14, 19, 1, 51], which facilitates the development of video grounding.”
Cascade Reasoning Network for Text-based Visual Question Answering (2020, background): “Relational reasoning has been widely explored in computer vision tasks [15], from region classification [9] to visual question answering [8, 48, 49], and further promoted after graph neural networks appeared [36].”
Breaking the Curse of Space Explosion: Towards Efficient NAS with Curriculum Search (2020, unknown): “9 595 4 NAT-DARTS (Guo et al., 2019) 73.”
Closed-Loop Matters: Dual Regression Networks for Single Image Super-Resolution (2020, methodology): “Deep neural networks (DNNs) have been the workhorse of many real-world applications, including image classification [18, 14, 9, 15, 27, 13], video understanding [46, 45,”