Deep Cascade Learning

By Enrique S. Marquez, Jonathon S. Hare, Mahesan Niranjan. 2018

126 citations recorded by the source corpus.

In this paper, we propose a novel approach for efficient training of deep neural networks in a bottom-up fashion using a layered structure. Our algorithm, which we refer to as deep cascade learning, is motivated by the cascade correlation approach of Fahlman and Lebiere, who introduced it in the context of perceptrons. We demonstrate our algorithm on networks of convolutional layers, though its applicability is more general. Such training of deep networks in a cascade directly circumvents the well-known vanishing gradient problem by ensuring that the output is always adjacent to the layer being trained. We present empirical evaluations comparing our deep cascade training with standard end–end training using back propagation of two convolutional neural network architectures on benchmark image classification tasks (CIFAR-10 and CIFAR-100). We then investigate the features learned by the approach and find that better, domain-specific, representations are learned in early layers when compared to what is learned in end–end training. This is partially attributable to the vanishing gradient problem that inhibits early layer filters to change significantly from their initial settings. While both networks perform similarly overall, recognition accuracy increases progressively with each added layer, with discriminative features learned in every stage of the network, whereas in end–end training, no such systematic feature representation was observed. We also show that such cascade training has significant computational and memory advantages over end–end training, and can be used as a pretraining algorithm to obtain a better performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

In Meta & Continual Learning

Citations in the corpus

Decoupled Greedy Learning of CNNs (2019, background): “Recently, several authors have revisited the classic (Ivakhnenko & Lapa, 1965; Bengio et al., 2007) approach of supervised greedy layer-wise training of neural networks (Huang et al., 2018; Marquez et al., 2018).”
Greedy Layerwise Learning Can Scale to ImageNet (2018, background): “Similarly (Kulkarni & Karande, 2017; Marquez et al., 2018) recently revisit layerwise training, but consider limited experimental settings.”
Training Neural Networks with Local Error Signals (2019, methodology): “Using supervised layer-wise loss functions, without fine-tuning, has also been explored previously (Mostafa et al., 2017; Malach & Shalev-Shwartz, 2018; Marquez et al., 2018).”
A Gentle Introduction to Deep Learning for Graphs (2019, background): “e eﬀectively propagated across layers and hence across node states. In supervised contexts, the constructive technique allows us to automatically determine the number of layers needed to solve a task [35, 86]. As explained in Figure 4, this characteristic is also related to the context needed by the”