Forward Thinking: Building and Training Neural Networks One Layer at a Time

By Chris Hettinger, Tanner Christensen, Ben Ehlert, J. Humpherys, Tyler J. Jarvis, Sean Wade. 2017

50 citations recorded by the source corpus.

We present a general framework for training deep neural networks without backpropagation. This substantially decreases training time and also allows for construction of deep networks with many sorts of learners, including networks whose layers are defined by functions that are not easily differentiated, like decision trees. The main idea is that layers can be trained one at a time, and once they are trained, the input data are mapped forward through the layer to create a new learning problem. The process is repeated, transforming the data through multiple layers, one at a time, rendering a new data set, which is expected to be better behaved, and on which a final output layer can achieve good performance. We call this forward thinking and demonstrate a proof of concept by achieving state-of-the-art accuracy on the MNIST dataset for convolutional neural networks. We also provide a general mathematical formulation of forward thinking that allows for other types of deep learning problems to be considered.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Optimization & Training

Citations in the corpus

Why Should We Add Early Exits to Neural Networks? (2020, methodology): “The advantage of this method is that, since each layer is relatively small (depending on the size of each f i and c i ), regularization problems such as vanishing and exploding gradients should be less prominent under this training approach [ 23 ] .”
Layerwise learning for quantum neural networks (2020, background): “In the classical setting, layerwise learning strategies have been shown to produce results comparable to training a full network with respect to error rate and time to convergence for classical NNs [43, 44].”
Differentiable Branching In Deep Networks for Fast Inference (2020, background): “Main contributions: While training objectives for the auxiliary classiﬁers have been explored in the literature [11, 15, 16], the problem of eﬃciently deciding when to exit has received considerable less attention.”