Occam's Razor

By Anselm Blumer, Andrzej Ehrenfeucht, David Haussler, Manfred K. Warmuth. 1987

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Philosophy

Citations in the corpus

Reconciling modern machine-learning practice and the classical bias–variance trade-off (2018, background): “Choosing the smoothest function that perfectly ﬁts observed data is a form of Occam’s razor: the simplest explanation compatible with the observations should be preferred (cf. [38, 6]).”
Learning in the presence of malicious errors (1988, methodology): “We begin by giving a generalization of Occam's Razor [4] for the case when errors are present in the examples.”
Implicit Regularization in Deep Learning (2017, background): “ f(x i;g(x i) jx i2Wg. The VC-dimension of many known model classes is a linear or low-degree polynomial of the number of parameters. The following generalization bound then holds with probability 1 [18,19]: L(f) L^(f) + O r VC-dim(F)lnm+ ln(1=) m ! (3.1.4) Feedforward Networks The VC dimension of f”
Learning higher-order logic programs (2019, background): “…in , and a maximum program size n u , unabstracted MIL has sample complexity: Proof According to the Blumer bound, which appears as a reformulation of Lemma 2.1 in [ 2 ] , the error of consistent hypotheses is bounded by ε with probability at least , where | H | is the size of the hypothesis space.”
Suboptimal behavior of Bayes and MDL in classification under misspecification (2004, methodology): “ quite small - see Section 5.1 and Figure 1. The answer to (3) is that there do exist simple algorithms which are consistent. An example is the algorithm which minimizes the Occam’s Razor bound (ORB) [7], Section 4.2. Theorem 1. (ORB consistency) For all priors P nonzero on a set of classiﬁers C, fo”
Deep learning generalizes because the parameter-function map is biased towards simple functions (2018, background): “For instance, the concepts of minimum description length (MDL) Rissanen (1978), Blumer algorithms Blumer et al. (1987); Wolpert & Waters (1994), and universal induction Ming & Vitányi (2014) all rely on a bias towards simple hypotheses.”
Relating Data Compression and Learnability (2003, methodology): “Note that the rst bound in the max expression is exactly the bound proven in [ BEHW87 ] which is the case where the kernel is empty.”