Convolutional Neural Fabrics

By Saurabh Saxena, Jakob Verbeek. 2016

232 citations recorded by the source corpus.

Despite the success of CNNs, selecting the optimal architecture for a given task remains an open problem. Instead of aiming to select a single optimal architecture, we propose a "fabric" that embeds an exponentially large number of architectures. The fabric consists of a 3D trellis that connects response maps at different layers, scales, and channels with a sparse homogeneous local connectivity pattern. The only hyper-parameters of a fabric are the number of channels and layers. While individual architectures can be recovered as paths, the fabric can in addition ensemble all embedded architectures together, sharing their weights where their paths overlap. Parameters can be learned using standard methods based on back-propagation, at a cost that scales linearly in the fabric size. We present benchmark results competitive with the state of the art for image classification on MNIST and CIFAR10, and for semantic segmentation on the Part Labels dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Neural Architecture Search & AutoML

Citations in the corpus

AutoGrow: Automatic Layer Growing in Deep Convolutional Networks (2019, neutral): “utogrow. time. For example, on CIFAR10 dataset, NAS used 22;400 Nvidia K40 GPU hours [39] and neural evolution [28] took 60;000 GPU hours [8] to perform the search. To accelerate NAS, one-shot models [29,26,2], DARTS [22] and NAS with Transferable Cell [40,20] were proposed. The search time reduces ”
Squeeze-and-Excitation Networks (2017, background): “Topology selection as a path through a fabric of possible designs [37] and direct architecture prediction [38], [39] have been proposed as additional viable architecture search tools.”
Neural Architecture Search with Reinforcement Learning (2016, background): “Hyperparameter optimization is an important research topic in machine learning, and is widely used in practice (Bergstra et al., 2011; Bergstra & Bengio, 2012; Snoek et al., 2012; 2015; Saxena & Verbeek, 2016).”
Learning Transferable Architectures for Scalable Image Recognition (2017, methodology): “ Related Work The proposed method is related to previous work in hyperparameter optimization [44, 4, 5, 54, 55, 6, 40] – especially recent approaches in designing architectures such as Neural Fabrics [48], DiffRNN [41], MetaQNN [3] and DeepArchitect [43]. A more ﬂexible class of methods for designin”
Regularized Evolution for Image Classifier Architecture Search (2018, methodology): “Some methods even forewent the idea of independent architectures (Saxena and Verbeek 2016).”
DARTS: Differentiable Architecture Search (2018, background): “The idea of searching architectures within a continuous domain is not new (Saxena and Verbeek, 2016; Ahmed and Torresani, 2017; Shin et al., 2018), but there are several major distinctions.”
Large-Scale Evolution of Image Classifiers (2017, background): “Also related to this work is that of Saxena & Verbeek (2016), who embed convolutions with different parameters into a species of “super-network” with many parallel paths.”
AutoML: A Survey of the State-of-the-Art (2019, background): “Many attempts ( [10], [111]–[115]) have been made to solve the problems encountered in the discrete samplebased approach.”
A Comprehensive Survey of Neural Architecture Search (2020, background): “In addition, these works [70–72] are also exploring the search for network architecture on continuous domains.”
Understanding and Robustifying Differentiable Architecture Search (2019, background): “…the search costs by, e.g., employing network morphisms [Cai et al., 2018a,b, Elsken et al., 2017, 2019b], weight sharing within one-shot models [Saxena and Verbeek, 2016, Bender et al., 2018, Pham et al., 2018] or multi-ﬁdelity optimization [Baker et al., 2017b, Falkner et al., 2018, Li et…”
BayesNAS: A Bayesian Approach for Neural Architecture Search (2019, background): “…as different subgraphs of a supergraph (the one-shot model) and shares weights between architectures that have edges of this super-graph in common (Saxena & Verbeek, 2016; Brock et al., 2018; Pham et al., 2018; Bender et al., 2018; Liu et al., 2019b; Cai et al., 2019; Xie et al., 2019; Zhang et…”
A Survey on Neural Architecture Search (2019, background): “Most of these one-shot architectures are based on an over-parameterized network (Saxena and Verbeek, 2016; Pham et al., 2018; Liu et al., 2019b; Xie et al., 2019b; Casale et al., 2019).”
sharpDARTS: Faster and More Accurate Differentiable Architecture Search (2019, background): “well dened search space of reasonable layers, connections, outputs, and hyperparameters. In fact, architecture search can itself be dened in terms of hyperparameters[12] or as a graph search problem[27, 19,2,24]. Furthermore, once a search space is dened various tools can be brought to bear on the p”
Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation (2019, methodology): “In addition, [66] was evaluated on segmenting face images into 3 classes [35], whereas our models are evaluated on large-scale segmentation datasets such as Cityscapes [13], PASCAL VOC 2012 [15], and ADE20K [89].”
Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search (2018, methodology): “The idea of super net and stochastic super net is also used in Saxena & Verbeek (2016); Veniat & Denoyer (2017) to explore macro architectures of neural nets.”
Efficient Progressive Neural Architecture Search (2018, neutral): “e duration of the search and the number of used GPUs. Method GPUs Time Parameters Error (days) (millions) (%) ResNet [7] - - 1.7 6.43 DenseNet [9] - - 25.6 3.46 Super Nets [27] - - - 9.21 ConvFabrics [21] - - 21.2 7.43 SMASH [4] 1 1.5 16.0 4.03 QNAS [2] 10 10 11.2 6.92 NAS [30] 800 28 37.4 3.65 ENAS”
MaskConnect: Connectivity Learning by Gradient Descent (2018, neutral): “Saxena and Verbeek [33] introduced convolutional neural fabric which are learnable 3D trellises that locally connect response maps at different layers of a CNN.”