Between the Devil and the Deep Blue Sea: Tensions Between Scientific Judgement and Statistical Model Selection

By D. Navarro. Computational Brain & Behavior, 2018

98 citations recorded by the source corpus.

Discussions of model selection in the psychological literature typically frame the issues as a question of statistical inference, with the goal being to determine which model makes the best predictions about data. Within this setting, advocates of leave-one-out cross-validation and Bayes factors disagree on precisely which prediction problem model selection questions should aim to answer. In this comment, I discuss some of these issues from a scientific perspective. What goal does model selection serve when all models are known to be systematically wrong? How might “toy problems” tell a misleading story? How does the scientific goal of explanation align with (or differ from) traditional statistical concerns? I do not offer answers to these questions, but hope to highlight the reasons why psychological researchers cannot avoid asking them.

Subfields: Artificial Intelligence, Experimental and Cognitive Psychology, History and Philosophy of Science

View Paper: https://www.semanticscholar.org/paper/57ab6d50eedc9f31fac059b12ab0e8db40dd1d94

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Experimental and Cognitive Psychology

In History and Philosophy of Science

Citations in the corpus

How Computational Modeling Can Force Theory Building in Psychological Science (2020, methodology): “One of the only situations in which multiple theories can be distinguished in a formal setting is when they can make sense of the available data (e.g., Levering et al., 2019; see also, however, Cox & Shiffrin, in press; Navarro, 2019; Wills & Pothos, 2012).”
Bayesian Workflow (2020, methodology): “In the real world we need to take into account the limitations of humans and computers, and this should be included in models of science and models of statistics (Navarro, 2019, Devezer et al., 2020).”
Ten simple rules for the computational modeling of behavioral data (2019, methodology): “While not always possible, distinguishing between models on the basis of qualitative patterns in the data is always preferable to quantitative model comparison (Navarro, 2018).”