Exploring Randomly Wired Neural Networks for Image Recognition

By Saining Xie, Alexander Kirillov, Ross B. Girshick, Kaiming He. 2019

389 citations recorded by the source corpus.

Neural networks for image recognition have evolved through extensive manual design from simple chain-like models to structures with multiple wiring paths. The success of ResNets and DenseNets is due in large part to their innovative wiring plans. Now, neural architecture search (NAS) studies are exploring the joint optimization of wiring and operation types, however, the space of possible wirings is constrained and still driven by manual design despite being searched. In this paper, we explore a more diverse set of connectivity patterns through the lens of randomly wired neural networks. To do this, we first define the concept of a stochastic network generator that encapsulates the entire network generation process. Encapsulation provides a unified view of NAS and randomly wired networks. Then, we use three classical random graph models to generate randomly wired graphs for networks. The results are surprising: several variants of these random generators yield network instances that have competitive accuracy on the ImageNet benchmark. These results suggest that new efforts focusing on designing better network generators may lead to new breakthroughs by exploring less constrained search spaces with more room for novel design.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

DSNAS: Direct Neural Architecture Search Without Parameter Retraining (2020, background): “This is orthogonal to the random wiring solution, which focuses on graph topology search [25].”
NAS evaluation is frustratingly hard (2019, methodology): “As such, future research could investigate the optimal wiring at a global level: an interesting work in this direction is Xie et al. (2019a).”
DARTS+: Improved Differentiable Architecture Search with Early Stopping (2019, background): “…it should be noticed that many recent progresses of DARTS could use “early stop-ping” to achieve better results, and the proposed “early stop-ping” criteria could be applied to many other types of search spaces, including the RandWire search space [35], mobile convnets search space [30], etc.”
AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures (2019, positive): “Our work is also related to the recent RandWire network [29], which showed that randomly connecting a sufﬁcient number of convolutional layers creates architectures with high performance.”
A Survey on Neural Architecture Search (2019, background): “The work by Xie et al. (2019b) builds upon the work of Liu et al. (2019b), but has three major differences.”
Evaluating the Search Phase of Neural Architecture Search (2019, neutral): “This phenomenon was already noticed by Zela et al. (2018), and concurrently to our work by Li & Talwalkar (2019); Xie et al. (2019); Ying et al. (2019), but the analysis of its impact or its causes went no further.”
Single Path One-Shot Neural Architecture Search with Uniform Sampling (2019, background): “Some papers [27,25,19] even show that several the state-of-the-art NAS methods perform similarly to the random search.”
An Adaptive Random Path Selection Approach for Incremental Learning. (2019, methodology): “Our choice of random path generation as a mechanism to select an optimal path is mainly inspired by the recent works of [23], [24], [25].”
XNAS: Neural Architecture Search with Expert Advice (2019, methodology): “We used 46 initial channels so the total number of network FLOPs is below 600[M]3, similar to other ImageNet architectures with small computation regime [48].”
Graph-Based Continual Learning (2020, background): “, 2001), random graphs appear sparingly in the machine learning literature, perhaps more noticeably in neural architecture search (Xie et al., 2019).”
Best Practices for Scientific Research on Neural Architecture Search (2019, neutral): “We note that there are several recent papers that cast doubts on much of the work in the field of NAS (Sciuto et al., 2019; Li and Talwalkar, 2019; Xie et al., 2019; Yang et al., 2020), and these have led to serious scepticism of outsiders concerning NAS.”
Permutation Invariant Graph Generation via Score-Based Generative Modeling (2020, methodology): “, 2018a), creating computation graphs for architecture search (Xie et al., 2019), as well as research in network science (Watts and Strogatz, 1998; Albert and Barabási, 2002; Leskovec et al.”
Search for Better Students to Learn Distilled Knowledge (2020, methodology): “Since the neural network architecture space is discrete, the strategies such as random search [35], reinforcement learning[39], evolutionary methods [23] were applied to tackle the discrete optimization problem.”
Single-Path Mobile AutoML: Efficient ConvNet Design and NAS Hyperparameter Optimization (2019, background): “Comparison with random search : An increasing amount of recent methods appeal to the practicality of random search as a simple, parameter-free NAS alternative [42].”
AttentionNAS: Spatiotemporal Attention Cell Search for Video Classification (2020, background): “Inception [28], or automatically discovered [41,42,35].”
Improved Differentiable Architecture Search for Language Modeling and Named Entity Recognition (2019, background): “The earliest of these approaches use reinforcement learning (RL) to learn promising architectures in a discrete space (Zoph and Le, 2016), whereas others have successfully modeled the problem in a continuous manner (Liu et al., 2019; Xie et al., 2019b; Huang and Xiang, 2019).”