Generalization in a linear perceptron in the presence of noise

By Anders Krogh, John A. Hertz. 1992

Subfields: Scaling & Learning Theory, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Scaling & Learning Theory

In Statistical Learning Theory

Citations in the corpus

A Simple Weight Decay Can Improve Generalization (1991, background): “Equation (17) was derived in [10] in the context of a particular eigenvalue spectrum.”
Triple descent and the two kinds of overfitting: where and why do they appear? (2020, unknown): “Although double descent has only recently gained interest in the context of deep learning, a seemingly similar phenomenon has been well-known for several decades for simpler models such as least squares regression [21, 22, 23, 14, 15], and has recently been studied in more detail in an attempt to sh”
Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks (2020, background): “Our results are consistent with those previously obtained for the linear perceptron with a noisy target 25,49 , which is a special case of kernel regression with a white band-limited spectrum.”
Think Locally, Act Globally: Federated Learning with Local and Global Representations (2020, background): “We assume a student-teacher setting [25, 21], where the goal is to train a network fû with weights û ∈ Rd on a task whose target is produced by a teacher network fu (i.”
Dynamics of stochastic gradient descent for two-layer neural networks in the teacher–student setup (2019, methodology): “To test these hypotheses, we calculated g for linear neural networks [47, 48] with g(x) = x.”
Towards Understanding Regularization in Batch Normalization (2018, methodology): “For instance, random noise in the input layer for data augmentation has long been discovered equivalent to a weight decay method, in the sense that the inverse of the signal-to-noise ratio acts as the decay factor (Krogh & Hertz, 1992; Rifai et al., 2011).”
If deep learning is the answer, what is the question? (2020, background): “4d) or when the number of data samples and model parameters both diverge towards infinity but their ratio is finite (the high-dimensional limit) 144,145 (Fig.”