AdaNet: Adaptive Structural Learning of Artificial Neural Networks

By Corinna Cortes, X. Gonzalvo, Vitaly Kuznetsov, Mehryar Mohri, Scott Yang. 2016

301 citations recorded by the source corpus.

We present new algorithms for adaptively learning artificial neural networks. Our algorithms (AdaNet) adaptively learn both the structure of the network and its weights. They are based on a solid theoretical analysis, including data-dependent generalization guarantees that we prove and discuss in detail. We report the results of large-scale experiments with one of our algorithms on several binary classification tasks extracted from the CIFAR-10 dataset. The results demonstrate that our algorithm can automatically learn network structures with very competitive performance accuracies when compared with those achieved for neural networks found by standard approaches.

Subfields: Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

AutoGrow: Automatic Layer Growing in Deep Convolutional Networks (2019, critical): “Some NAS methods [2, 6, 23] can find DNNs with different depths, however, the maximum depth is pre-defined and shallower depths are obtained by padding zero operations or selecting shallower branches, while our AutoGrow learns the depth in an open domain to find a minimum depth, beyond which no accu”
Why Should We Add Early Exits to Neural Networks? (2020, methodology): “Only recently, however, have they become a successful tool for training them, e.g., through boosting [ 18, 25 ] , clustering [ 47 ] , and kernel similarity [ 35 ] .”
Progressive Neural Architecture Search (2017, positive): “Finally, [24,25] grow CNNs sequentially using boosting.”
Regularized Evolution for Image Classifier Architecture Search (2018, methodology): “Other methods have also been applied: cascadecorrelation (Fahlman and Lebiere 1990), boosting (Cortes et al. 2017), hill-climbing (Elsken, Metzen, and Hutter 2017), MCTS (Negrinho and Gordon 2017), SMBO (Mendoza et al.”
BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search (2019, background): “Since the popular NASNet paper [54], there has been a flurry of research on neural architecture search [27, 34, 29, 17, 6, 15, 36, 3, 1, 28].”
Deep Active Learning with a Neural Architecture Search (2018, methodology): “When considering NAS for fully-connected networks, [4] proposed an algorithm that iteratively adds neurons to an existing layer or to initiate a new layer.”
Evolutionary-Neural Hybrid Agents for Architecture Search (2018, neutral): “…deep RL and evolution, different approaches have been applied to architecture search and hyperparameter optimization: bayesian optimization [15], cascade-correlation [14], boosting [9], deep-learning based tree search [30], hill-climbing [13], model-based approaches [21], and random search [4].”
Auto-Meta: Automated Gradient Based Meta Learner Search (2018, background): “This line of thinking is present in hypernetworks [4], co-evolving NEAT [15], boosting [6, 11], MCTS [17], early stopping of unpromising models [3], and progressive neural architecture search(PNAS) [13].”
Neural Architecture Search with Bayesian Optimisation and Optimal Transport (2018, background): “A related line of work first sets up a search space for architectures via incremental modifications, and then explores this space via random exploration, MCTS, or A* search [6, 22, 29].”
Lifelong Learning with Dynamically Expandable Networks (2017, methodology): “Cortes et al. (2016) also propose a network that can adaptively learn both the structure and the weights to minimize the given loss, based on boosting theory.”
Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learning (2017, methodology): “These include techniques based on reinforcement learning (Zoph & Le, 2017), (Baker et al., 2017), evolutionary algorithms (Miikkulainen et al., 2017), approximate random simulations (Brock et al., 2017), and adaptive growth (Cortes et al., 2016).”
Routing Networks and the Challenges of Modular and Compositional Computation (2019, methodology): “These approaches have been learned using reinforcement learning [Zoph and Le, 2017, Baker et al., 2017], evolutionary algorithms [Miikkulainen et al., 2017, Fernando et al., 2017], approximate random simulations [Brock et al., 2017], and adaptive growth [Cortes et al., 2016].”
Learning Deep ResNet Blocks Sequentially using Boosting Theory (2017, methodology): “3 which follows directly from Lemma 2 of (Cortes et al., 2016).”
Automated Machine Learning: From Principles to Practices (2018, background): “In the disciplines of earth and environmental sciences, AutoML has demonstrated its efficacy in characterizing unconventional reservoirs, accurately predicting key properties like shale volume, porosity, and bitumen content [99].”
Functional Gradient Boosting based on Residual Network Perception (2018, unknown): “Cortes et al. (2017) proposed AdaNet to adaptively learn both the structure of the network and its weight, and provided data-dependent generalization guarantees for an adaptively learned network; however, the learning strategy quite differs from our method and the convergence rate is unclear.”
Class-incremental Learning with Pre-allocated Fixed Classifiers (2020, methodology): “[28] exploits boosting algorithm to control network architecture growth balancing its complexity with empirical risk minimization.”
A Survey of Deep Learning: Platforms, Applications and Emerging Research Trends (2018, methodology): “[29] developed AdaNet, a set of algorithms to adaptiveley learn ANN network structure and weights utilizing explicit Rademacher complexity measures to define data-dependent learning and generalisation bounds.”
Fast Task-Aware Architecture Inference (2019, methodology): “• Morphing methods [7, 3] start with an initial model architecture and they iteratively refine the architecture during training until a certain objective is hit (e.”