Accelerated Learning in Layered Neural Networks

By Sara A. Solla, Esther Levin, M. Fleisher. 1988

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Deep learning in neural networks: An overview (2014, methodology): “…1970) are computationally too expensive for large NNs. Partial BFGS (Battiti, 1992; Saito & Nakano, 1997) and conjugate gradient (Hestenes & Stiefel, 1952; Møller, 1993) as well as other methods (Cauwenberghs, 1993; Schmidhuber, 1989a; Solla, 1988) provide sometimes useful fast alternatives.”
Supervised Contrastive Learning (2020, background): “learning and supervised learning. Due to the large amount of literature, we focus on the most relevant papers. The cross-entropy loss was introduced as a powerful loss function to train deep networks [38,1,28]. The key idea is simple and intuitive: each class is assigned a target (usually 1-hot) vec”
Neural networks for the prediction and forecasting of water resource variables: a review of modelling issues and applications (2000, methodology): “The mean squared error (MSE) function is most commonly used, but other error functions have also been proposed (e.g. Solla et al., 1988).”
Exact Calculation of the Hessian Matrix for the Multilayer Perceptron (1992, background): “…derivatives of the error become ∂E p ∂z m = (z m − t m ) ∂ 2 E p ∂z 2 m = 1 (23) Another commonly used error measure is the relative entropy (Solla et al., 1988) defined by ˆ E p = m {t m ln z m + (1 − t m ) ln(1 − z m )} (24) The derivatives ofˆE p take a particularly elegant form when the…”
A Tutorial on Energy-Based Learning (2006, background): “In the neural network classification literature, it is known as the cross-entropy loss [57].”
Large-Scale Multi-label Text Classification - Revisiting Neural Networks (2013, methodology): “e c. In this paper we use c= 0. Note that it has been pointed out that the slope of the cost function as a function of the parameters plays an important role in learning parameters of neural networks [19, 10]. Figure 2 shows different shapes of the functions and slope steepness. We conjecture that t”
RASR/NN: The RWTH neural network toolkit for speech recognition (2014, methodology): “Currently, RASR provides the commonly used activation functions sigmoid, tanh, linear and softmax as well as the recently proposed rectified linear units (rlu) [11].”
Natural Language Grammatical Inference with Recurrent Neural Networks (2000, background): “The error at each point in the graphs is the NMSE over the complete training set.”
The Evidence Framework Applied to Classification Networks (1992, background): “It is well known that the natural objective function in this case is an information{based distance measure, rather than the sum of squared errors (Bridle, 1989, Hinton and Sejnowski, 1986, Hopeld, 1987, Solla et al., 1988).”
Understanding the difficulty of training deep feedforward neural networks (2010, background): “This is not a new observation (Solla et al., 1988) but we ﬁnd it important to stress here.”