AdaNet: Adaptive Structural Learning of Artificial Neural Networks

2017

Subfields: Neural Architecture Search & AutoML

Citations in the corpus

• AutoGrow: Automatic Layer Growing in Deep Convolutional Networks (2019, neutral): “AdaNet [6] can adapt neural networks to different datasets.”
• Deep Active Learning with a Neural Architecture Search (2018, methodology): “When considering NAS for fully-connected networks, [4] proposed an algorithm that iteratively adds neurons to an existing layer or to initiate a new layer.”
• Auto-Meta: Automated Gradient Based Meta Learner Search (2018, background): “This line of thinking is present in hypernetworks [4], co-evolving NEAT [16], boosting [6, 11], MCTS [18], early stopping of unpromising models [3], and progressive neural architecture search (PNAS) [14].”
• Neural Architecture Search with Bayesian Optimisation and Optimal Transport (2018, methodology): “A related line of work first sets up a search space for architectures via incremental modifications, and then explores this space via random exploration, MCTS, or A* search [6, 25, 32].”
• Lifelong Learning with Dynamically Expandable Networks (2017, background): “Cortes et al. (2016) also propose a network that can adaptively learn both the structure and the weights to minimize the given loss, based on boosting theory.”
• Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learning (2017, methodology): “These include techniques based on reinforcement learning (Zoph & Le, 2017), (Baker et al., 2017), evolutionary algorithms (Miikkulainen et al., 2017), approximate random simulations (Brock et al., 2017), and adaptive growth (Cortes et al., 2016).”
• Greedy Layerwise Learning Can Scale to ImageNet (2018, background): “Similarly, Cortes et al. (2016) recently proposed a progressive learning method that builds a network such that the architecture can adapt to the problem, with theoretical contributions to structure learning, but not on problems where deep networks are unmatched in performance.”
• Fast Task-Aware Architecture Inference (2019, methodology): “• Morphing methods [7, 3] start with an initial model architecture and they iteratively refine the architecture during training until a certain objective is hit (e.g., model size or flops per inference).”
• AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data (2020, methodology): “The internals of this framework thus remain unclear, although it is known to at least utilize both tree ensembles and the AdaNet method of boosting neural network ensembles (Cortes et al., 2017).”
• Adaptive Neural Trees (2018, methodology): “…of NNs (Fahlman & Lebiere, 1990; Hinton et al., 2006; Xiao et al., 2014; Chen et al., 2016; Srivastava et al., 2015; Lee et al., 2017; Cai et al., 2018; ˙Irsoy & Alpaydın, 2018), or more broadly, the field of neural architecture search (Zoph & Le, 2017; Brock et al., 2017; Cortes et al., 2017).”
• Accelerating Neural Architecture Search using Performance Prediction (2017, methodology): “With the advent of large scale automated architecture search [1, 14, 4, 8, 15, 3], methods such as ours will be vital in exploring large and complex search spaces.”
• Learning Intrinsic Sparse Structures within Long Short-term Memory (2017, methodology): “Philipp & Carbonell (2017) uses group Lasso regularization to dynamically add and eliminate neurons in FNNs. Cortes et al. (2017) incrementally adds sub-networks to current networks by incrementally reducing the objective function.”
• TabTransformer: Tabular Data Modeling Using Contextual Embeddings (2020, background): “These include Deep & Cross Networks (Wang et al. 2017), Wide & Deep Networks (Cheng et al. 2016), TabNets (Arik and Pfister 2019), and AdaNet (Cortes et al. 2016).”
• Patch-based Knowledge Distillation for Lifelong Person Re-Identification (2022, background): “Expansion-based methods [11, 41, 51] handle increasing knowledge by expanding model architecture on demand.”
• TabNet: Attentive Interpretable Tabular Learning (2019, methodology): “AutoML Tables (denoted as (cid:63)(cid:63) ) is an automated machine learning development tool based on ensemble of models including linear feed-forward deep neural network, gradient boosted decision tree, AdaNet (Cortes et al., 2016) and ensembles (AutoML, 2019).”
• Functional Gradient Boosting based on Residual Network Perception (2018, unknown): “Cortes et al. (2017) proposed AdaNet to adaptively learn both the structure of the network and its weight, and provided data-dependent generalization guarantees for an adaptively learned network; however, the learning strategy quite differs from our method and the convergence rate is unclear.”