Neural Networks and the Bias/Variance Dilemma

By Stuart Geman, Elie Bienenstock, René Doursat. 1992

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Scaling & Learning Theory

Citations in the corpus

Reconciling modern machine-learning practice and the classical bias–variance trade-off (2018, background): “Conventional wisdom in machine learning suggests controlling the capacity of the function class H based on the bias-variance trade-oﬀ by balancing under-ﬁtting and over-ﬁtting (cf., [17, 21]): 1.”
Deep learning in neural networks: An overview (2014, background): “Most approaches address the bias/variance dilemma (Geman et al., 1992) through strong prior assumptions.”
Relational inductive biases, deep learning, and graph networks (2018, background): “Inductive biases often trade flexibility for improved sample complexity and can be understood in terms of the bias-variance tradeoff (Geman et al., 1992).”
Practical Recommendations for Gradient-Based Training of Deep Architectures (2012, background): “Multi-Task Learning and Multi-Relational Learning Parameter sharing (Lang and Hinton, 1988; LeCun, 1989; Lang and Hinton, 1988; Caruana, 1993; Baxter, 1995, 1997) be decomposed as the sum of a bias term and a variance term (Geman et al. , 1992).”
Active Learning with Statistical Models (1996, background): “The expectation inside the integral may be decomposed as follows (Geman, Bienenstock, & Doursat, 1992): (2) where ED[(cid:1)] denotes the expectation over training sets D and the remaining expectations on the right-hand side are expectations with respect to the conditional density P(yjx).”
Rethinking eliminative connectionism. (1998, neutral): “Simulations by Geman, Bienstock, and Doursat (1992), Pavel, Gluck, and Henkle (1988), Pazzani and Dyer (1987), Prasada and Pinker (1993), De- Losh, Busemeyer, and McDaniel (1997), and Busemeyer, McDaniel, and Byun (1997) are consistent with the suggestion of Denker and others that eliminative…”
Sequence Learning (2001, unknown): “a neural network into Bias and Variance [26].”
Active Learning Literature Survey (2009, background): “They take advantage of the result by Geman et al. (1992) showing that a learner’s expected future generalization error can be decomposed in the following way:”
Simple Heuristics That Make Us Smart (1999, background): “This failure of generalization, a phenomenon known as overfitting (e.g., Geman et al., 1992; Massaro, 1988), stems from assuming that every detail is of utmost relevance.”
Neural Network Ensembles, Cross Validation, and Active Learning (1994, background): “Equation (10) expresses the tradeoff between bias and variance in the ensemble, but in a different way than the the common bias-variance relation [4] in which the averages are over possible training sets instead of ensemble averages.”
Lifelong Learning: A Case Study. (1995, background): “Since deducing the output of unseen, future data is impossible without making further assumptions [31] [68] [19] [73], every learning algorithm makes inherent assumptions concerning the nature of the data.”
When Neurons Fail (2017, background): “This creates a dilemma that somehow resembles the famous bias/variance dilemma [35] in machine learning.”