Effective degrees of freedom: a flawed metaphor

By L. Janson, William Fithian, Trevor J. Hastie. 2013

67 citations recorded by the source corpus.

Summary To most applied statisticians, a fitting procedure’s degrees of freedom is synonymous with its model complexity, or its capacity for overfitting to data. In particular, it is often used to parameterize the bias-variance tradeoff in model selection. We argue that, on the contrary, model complexity and degrees of freedom may correspond very poorly. We exhibit and theoretically explore various fitting procedures for which degrees of freedom is not monotonic in the model complexity parameter, and can exceed the total dimension of the ambient space even in very simple settings. We show that the degrees of freedom for any non-convex projection method can be unbounded.

Subfields: Management Science and Operations Research, Statistics and Probability

Citations in the corpus

• Extended Comparisons of Best Subset Selection, Forward Stepwise Selection, and the Lasso (2017, neutral): “2012), whereas the degrees of freedom of both forward stepwise and best subset selection can greatly exceed k at any given step k ≥ 1 (Kaufman and Rosset, 2014; Janson et al., 2015).”
• Degrees of freedom and model search (2014, unknown): “, see Figure 4 of Tibshirani & Knight (1999) for an early example, and Figure 2 of Janson et al. (2013) for a recent example).”