Modular Learning in Neural Networks

By Dana H. Ballard. 1987

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Deep learning in neural networks: An overview (2014, methodology): “It proposed unsupervised AE hierarchies (Ballard, 1987), closely related to certain post-2000 feedforward Deep Learners based on UL (Section 5.15).”
A practical tutorial on autoencoders for nonlinear feature fusion: Taxonomy, models, software and guidelines (2018, methodology): “AEs were ﬁrstly introduced [15] as a way of conducting pretraining in ANNs.”
Continual Learning via Local Module Composition (2021, background): “Modularity in neural networks is studied in the context of scalability [9], and more recently as a way to achieve compositionality and systematic generalization [6, 47, 15, 8, 27, 19] as well as for multi-task learning [65, 82].”
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (2015, background): “Today’s faster computers, such as GPUs, mitigate the Fundamental Deep Learning Problem for FNNs [181, 34, 198, 38, 40].”
Self-Supervised Learning: Generative or Contrastive (2020, background): “Autoencoder (AE) was first introduced in [8] for pre-training artificial neural networks.”
Natural Language Processing With Modular PDP Networks and Distributed Lexicon (1991, background): “A complex cognitive task can often be broken down into simpler subtasks, which are performed sequentially, one task depending on the output of another (Ballard, 1987; Minsky, 1985).”
Structural Autoencoders Improve Representations for Generation and Transfer (2020, methodology): “Here deep learning provides a ﬂexible paradigm to train complex architectures based on autoencoders [6, 7] allowing us to embed powerful inductive biases into our models to further structure the representations making them more useful.”
My First Deep Learning System of 1991 + Deep Learning Timeline 1962-2013 (2013, background): “In 1987, Ballard published ideas on unsupervised autoencoder hierarchies [7], related to post-2000 feed-forward Deep Learners (TL 1.9) based on unsupervised pre-training, e.g., [44]; compare survey [45] and somewhat related RAAMs [69].”
Self-Supervised Learning in Remote Sensing: A review (2022, background): “AUTOENCODER The concept of AEs was introduced as early as in [54] to serve as pretraining of artificial neural networks.”
Multimodal deep learning for biomedical data fusion: a review (2022, background): “AEs are architectures that are able to learn such embeddings z from input x through an encoder function f(x) and a decoder function g(z) in an unsupervised manner (see Figure 3b) [15].”
Deep Learning for Medical Anomaly Detection – A Survey (2020, methodology): “Auto Encoders (AEs) Since their introduction in [18] as a method for pre-training deep neural networks, AEs have been widely used as a method for automatic feature learning [19].”
Deep Learning for EMG-based Human-Machine Interaction: A Review (2021, background): “The bidirectional RNN (Bi-RNN) [25] can resolve this question by stacking two RNNs in the forward and backward directions together, to make decisions that are based on not only previous states but also future states.”
Metabolomics and Multi-Omics Integration: A Survey of Computational Methods and Resources (2020, methodology): “Autoencoders are another type of unsupervised learning method used in omics data exploration; they are used for learning a set of latent variables that can be used to reconstruct data [78].”
Connectionist Learning Procedures (1989, methodology): “An improvement on this strictly bottom-up scheme is described by Ballard [11].”
Machine Learning and Deep Learning Algorithms for Bearing Fault Diagnostics - A Comprehensive Review (2019, methodology): “Auto-encoder is proposed in the 1980s as an unsupervised pre-training method for ANNs [119], [120].”
Nonlinear principal component analysis using autoassociative neural networks (1991, methodology): “Training to learn the identity mapping has been called self-supervised backpropagation or autoassociation (Ballard, 1987; Cottrell et al., 1987), and has been used to solve so-called " encoder/decoder " problems, discussed in the subsection on Relationship to Other Work.”
Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks (2020, background): “Modularity is an important organization principle in both artiﬁcial (Ballard, 1987; Baldwin & Clark, 2000) and biological (von Dassow & Munro, 1999; Lorenz et al., 2011; Clune et al., 2013) systems.”
Modular Neural Networks and Self-Decomposition (1997, background): “Modularity has also the advantage to enable a localisation of the computation in the whole system (Martinez 1992) (Hrycej 1992) (Jacobs et al.”