Knowledge Matters: Importance of Prior Information for Optimization

By Caglar Gulcehre, Yoshua Bengio. 2013

173 citations recorded by the source corpus.

We explore the effect of introducing prior information into the intermediate level of neural networks for a learning task on which all the state-of-the-art machine learning algorithms tested failed to learn. We motivate our work from the hypothesis that humans learn such intermediate concepts from other individuals via a form of supervision or guidance using a curriculum. The experiments we have conducted provide positive evidence in favor of this hypothesis. In our experiments, a two-tiered MLP architecture is trained on a dataset with 64x64 binary inputs images, each image with three sprites. The final task is to decide whether all the sprites are the same or one of them is different. Sprites are pentomino tetris shapes and they are placed in an image with different locations using scaling and rotation transformations. The first part of the two-tiered MLP is pre-trained with intermediate-level targets being the presence of sprites at each location, while the second part takes the output of the first part as input and predicts the final task's target binary event. The two-tiered MLP architecture, with a few tens of thousand examples, was able to learn the task perfectly, whereas all other algorithms (include unsupervised pre-training, but also traditional algorithms like SVMs, decision trees and boosting) all perform no better than chance. We hypothesize that the optimization difficulty involved when the intermediate pre-training is not performed is due to the {\em composition} of two highly non-linear tasks. Our findings are also consistent with hypotheses on cultural learning inspired by the observations of optimization problems with deep learning, presumably because of effective local minima.

Subfields: Computer Vision and Pattern Recognition, Cultural Studies, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Cultural Studies

In Optimization & Training

Citations in the corpus

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift (2015, methodology): “On the contrary, (Gülçehre & Bengio, 2013) apply the standardization layer to theoutputof the nonlinearity, which results in sparser activations.”
Network In Network (2013, methodology): “For example, the Structured Multilayer Perceptron (SMLP) [9] applies a shared multilayer perceptron on different patches of the input image; in another work, a neural network based filter is trained for face detection [10].”
What can the brain teach us about building artificial intelligence? (2016, background): “This sort of richer feedback can easily be incorporated into neural networks, and doing so can enhance learning (Gulcehre and Bengio, 2016).”
A practical tutorial on autoencoders for nonlinear feature fusion: Taxonomy, models, software and guidelines (2018, neutral): “Although they have been successfully used in [47, 48], the authors had to resort to a few detours.”
Learning and Refining of Privileged Information-Based RNNs for Action Recognition from Depth Sequences (2017, background): “[4] explore the architecture by providing intermediate targets.”
Learned-Norm Pooling for Deep Feedforward and Recurrent Neural Networks (2013, methodology): “This is the best result reported so far on Pentomino dataset [14] without using any kind of prior information about the task (the best previous result was 44.6% error).”
Breaking the Curse of Space Explosion: Towards Efficient NAS with Curriculum Search (2020, unknown): “The key idea follows the basic concept of curriculum learning that humans and animals can learn much better when they gradually learn new knowledge (Bengio et al., 2009; Gülçehre & Bengio, 2016).”
Batch-normalized joint training for DNN-based distant speech recognition (2016, methodology): “As shown previously [37], such prior knowledge becomes progressively more precious as the complexity of the problem increases and can thus be very helpful for a distant speech recognition task.”
Learning from Multiple Teacher Networks (2017, background): “to the intermediate outputs and [11, 20, 30] argued that the discriminability of the internal representations against the labels would make a contribution to the training of deep networks.”
Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, methodology): “ction2) on the transformed data X. NRR is used to recover the possible reduced representation capacity. Take BN as an example. BN [8] considers each spatial position in a feature map as a sample [8], [66] and the NAP is: X = BN(X) 2Rd mhw; (20) which means that the statistics are calculated along th”
Modelling, Visualising and Summarising Documents with a Single Convolutional Neural Network (2014, methodology): “This architecture is inspired by Gulechere and Bengio [7] who show that learning appropriate intermediate representations helps generalisation, and also by Hinton et al. [10] who show that by forcing information to pass through carefully chosen bottlenecks it is possible to control the types of…”
Fast and Accurate Sequence Labeling with Iterated Dilated Convolutions (2017, background): “By doubling the dilation width at each layer, the size of the receptive field grows exponentially while the number of parameters grows only linearly with the number of layers, so a pixel representation quickly incorporates rich global evidence from the entire image.”
How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation (2014, background): “…continuous and gradients with respect to lower layers (through many non-linearities) appear less able to correctly train them, especially when considering deeper networks trying to learn more abstract concepts obtained from the composition of simpler ones (Bengio, 2009; Gulcehre and Bengio, 2013).”
Sub-Task Decomposition Enables Learning in Sequence to Sequence Tasks (2022, background): “Gülçehre & Bengio (2016), one of the earliest works on this subject, propose that supervision for the single-hop intermediate steps is crucial for avoiding bad local minima in the optimization of neural networks.”
Mollifying Networks (2016, background): “The task is to predict whether if there is a different shape in the image or not (Gülçehre and Bengio, 2013).”
Failures of Gradient-Based Deep Learning (2017, background): “Before continuing, we note that a conceptually similar experiment to ours has been reported in [13] (also involving a composition of an image recognition task and a simple Boolean formula, and with qualitatively similar results).”
Multimodal Transitions for Generative Stochastic Networks (2013, background): “…indicated by the success of the unsupervised layer-wise pre-training procedures [5], semi-supervised embedding [33], and intermediate-level hints [21]), i.e., that it may suffer less from the difﬁculty of propagating credit across a large network, which has been observed for supervised learning.”