Sensitivity and Specificity of Information Criteria
By John J. Dziak, D. Coffman, Stephanie T. Lanza, Runze Li, Lars S. Jermiin. bioRxiv, 2018
676 citations recorded by the source corpus.
Information criteria (ICs) based on penalized likelihood, such as Akaike’s Information Criterion (AIC), the Bayesian Information Criterion (BIC), and sample-size-adjusted versions of them, are widely used for model selection in health and biological research. However, different criteria sometimes support different models, leading to discussions about which is the most trustworthy. Some researchers and fields of study habitually use one or the other, often without a clearly stated justification. They may not realize that the criteria may disagree. Others try to compare models using multiple criteria but encounter ambiguity when different criteria lead to substantively different answers, leading to questions about which criterion is best. In this paper we present an alternative perspective on these criteria that can help in interpreting their practical implications. Specifically, in some cases the comparison of two models using ICs can be viewed as equivalent to a likelihood ratio test, with the different criteria representing different alpha levels and BIC being a more conservative test than AIC. This perspective may lead to insights about how to interpret the ICs in more complex situations. For example, AIC or BIC could be preferable, depending on the relative importance one assigns to sensitivity versus specificity. Understanding the differences and similarities among the ICs can make it easier to compare their results and to use them to make informed decisions.
Subfields: Artificial Intelligence, Statistics and Probability
View Paper: https://www.semanticscholar.org/paper/f605379a39f727dc29568f63fdf2f2454123296d
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Statistics and Probability
Citations in the corpus
- Regularized Gaussian Psychological Networks: Brief Report on the Performance of Extended BIC Model Selection (2016, background): “Setting γ = 0.25 improved the detection rate, but sometimes led to poorly estimated networks based on polychoric correlations. γ can be set to 0 to err more on the side of discovery (Dziak et al., 2012), but should be done with care in low sample polychoric correlation matrices.”
- A Tutorial on Regularized Partial Correlation Networks (2016, background): “In sum, the choice of the hyperparameter is somewhat arbitrary and up to the researcher, and depends on the relative importance assigned to caution or discovery (Dziak et al., 2012).”
- Generalized Network Psychometrics: Combining Network and Latent Variable Models (2016, neutral): “However, it is important to note that the choice of a particular information criterion cannot be argued by these numbers alone, and depends on the relative importance one assigns to the side of discovery (optimizing sensitivity) or the side of caution (optimizing specificity; Dziak et al. 2012).”
- Practitioner’s Guide to Latent Class Analysis: Methodological Considerations and Common Pitfalls. (2020, background): “Whereas, as n increases, the AIC has a tendency to select more complex models (more classes), as the best fitting, because sample size is not a determining factor in its estimation (51).”
- Exploratory graph analysis: A new approach for estimating the number of dimensions in psychological research (2016, background): “However, Dziak, Coffman, Lanza and Li [27] showed that BIC decreases its underestimation and increases its correctness in estimating the number of factors when the sample size is greater than 200 cases.”