Dynamic node creation in backpropagation networks

By Timur Ash. 1989

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Deep learning in neural networks: An overview (2014, neutral): “…(or biases towards simplicity) are implicit in constructive and pruning algorithms, e.g., layer-by-layer sequential network construction (e.g., Ash, 1989; Burgess, 1994; Fahlman, 1991; Fritzke, 1994; Gallant, 1988; Honavar & Uhr, 1988, 1993; Ivakhnenko, 1968, 1971; Moody, 1989; Parekh, Yang,…”
Learning the Number of Neurons in Deep Networks (2016, methodology): “In particular, a forerunner approach was presented in [Ash, 1989] to dynamically add nodes to an existing architecture.”
A Wholistic View of Continual Learning with Deep Neural Networks: Forgotten Lessons and the Bridge to Active and Open World Learning (2020, background): “For now, we will start with a typical categorization of existing deep continual works into the three categories of regularization, rehearsal and architectural approaches, in consistency ∙ LWF (LiandHoiem,2016) ∙ EBLL (Rannenetal.,2017) ∙ Pseudorehearsal (Robins,1995) ∙ Pseudo-recurrentnets (French,1”
NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune Paradigm (2017, background): “A constructive approach starts with a small network and iteratively adds connections/neurons [16].”
A New Adaptive Merging and Growing Algorithm for Designing Artificial Neural Networks (2009, background): “Dynamic neuron creation [41] is probably the first-ever constructive algorithm in designing”
Learning in Artificial Neural Networks: A Statistical Perspective (1989, methodology): “The search for an appropriate technique for determining network complexity has been the focus of considerable effort to date (e.g., Rumelhart 1988; Ash 1989; Hirose et al. 1989).”
Hybrid decision tree (2002, background): “Note that there are many other neural classifiers having the ability of dynamically adjusting network topology (Ash, 1989; Fahlman, 1990).”
Improving Generalization with Active Learning (1994, methodology): “The theory and methods described here should, with some modification, be equally applicable to other trainable classifiers.”
A constructive algorithm for training cooperative neural network ensembles (2003, background): “Although the criteria for node addition and halting network construction in CNNE appear to be similar to those used in some constructive algorithms for monolithic NNs [1], [20], [30], [36], [50], there is a significant difference.”
Generative learning structures and processes for generalized connectionist networks (1993, methodology): “, generative learning (Honavar & Uhr, 1988; 1989a; 1989b) and related approaches (Ash, 1989; Diederich, 1988; Fahlman & Lebiere, 1990; Hanson, 1990; Gallant, 1990; Nadal, 1989; Rujan & Marchand, 1989).”
Constructive algorithms for structure learning in feedforward neural networks for regression problems (1997, background): “Constructive algorithms in this category [36], [64], [77], [80], [84]–[88] are variants of the dynamic node creation (DNC) network proposed by Ash [84].”
Learning and development in neural networks: the importance of starting small. (1993, background): “Allowing networks to reconfigure dynamically or acquire additional nodes has been shown to facilitate learning (Ash, 1989; Fahlman & Lebiere, 1990; Shultz & Schmidt, 1991).”
An evolutionary algorithm that constructs recurrent neural networks (1994, background): “Constructive algorithms initially assume a simple network and add nodes and links as warranted [2-8], while destructive methods start with a large network and prune off superfluous components [9-12].”
Flat Minima (1997, unknown): “Examples are \sequential network construction" (Fahlman & Lebiere, 1990; Ash, 1989; Moody, 1989), input pruning (Moody, 1992; Refenes et al., 1994), unit pruning (White, 1989; Mozer & Smolensky, 1989; Levin et al., 1994), weight pruning, e.g. \optimal brain damage" (LeCun et al., 1990), \optimal…”
Many-Layered Learning (2002, unknown): “Several constructive methods add hidden units during learning, increasing the width of one or more existing layers (Ash, 1989; Hanson, 1990; Frean, 1990; Wynne-Jones, 1991; Utgoff & Precup, 1998).”
SUSTAIN: a network model of category learning. (2004, background): “…(Busemeyer & McDaniel, 1997; Karnin, 1990), whereas other adaptive architecture models (including SUSTAIN) begin small and expand as needed (Ash, 1989; Azimi-Sadjadi, Sheedvash, & Trujillo, 1993; Carpenter, Grossberg, & Reynolds, 1991; Cho, 1997; Fahlman & Lebiere, 1990; Kruschke &…”
A dynamic ensemble learning algorithm for neural networks (2019, methodology): “Next, apply a constructive algorithm [56] based on Ash’s [57] dynamic node creation method for the first (later on the odd number of NNs in sequence in the ensemble) NN training.”