Fearing the Future of Empirical Psychology: Bem's (2011) Evidence of Psi as a Case Study of Deficiencies in Modal Research Practice

By Etienne P LeBel, Kurt R. Peters. Review of General Psychology, 2011

145 citations recorded by the source corpus.

In this methodological commentary, we use Bem's (2011) recent article reporting experimental evidence for psi as a case study for discussing important deficiencies in modal research practice in empirical psychology. We focus on (a) overemphasis on conceptual rather than close replication, (b) insufficient attention to verifying the soundness of measurement and experimental procedures, and (c) flawed implementation of null hypothesis significance testing. We argue that these deficiencies contribute to weak method-relevant beliefs that, in conjunction with overly strong theory-relevant beliefs, lead to a systemic and pernicious bias in the interpretation of data that favors a researcher's theory. Ultimately, this interpretation bias increases the risk of drawing incorrect conclusions about human psychology. Our analysis points to concrete recommendations for improving research practice in empirical psychology. We recommend (a) a stronger emphasis on close replication, (b) routinely verifying the integrity of measurement instruments and experimental procedures, and (c) using stronger, more diagnostic forms of null hypothesis testing.

Subfields: Cognitive Neuroscience, Experimental and Cognitive Psychology, Social Psychology

View Paper: https://www.semanticscholar.org/paper/fee248259a9553982a91ac74f85385c3b37fd614

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Experimental and Cognitive Psychology

In Social Psychology

Citations in the corpus

The Replication Recipe: What Makes for a Convincing Replication? (2014, background): “…problems in science and is not a limitation of the Replication Recipe, but rather of the modal research practices in some areas of research (LeBel & Peters, 2011; Murayama, Pekrun, & Fiedler, in press; Simmons et al., 2011; SPSP Task Force on Research & Publication Practices, in press).”
Pre-registration in social psychology—A discussion and suggested template (2016, methodology): “…in psychology are often subjective, there is a temptation to choose, out of many possible analyses, the one that gives the most consistent or significant results, and to dismiss as exploratory or flawed those elements of a study that fail to achieve the desired e↵ect (LeBel & Peters, 2011).”
Registered Reports A Method to Increase the Credibility of Published Results (2014, background): “However, conceptual replication are not best suited to clarify the truth of any particular effect because nonsignificant findings are attributable to changes in the research design, and rarely lead researchers to question the phenomenon (LeBel & Peters, 2011; Nosek, Spies, & Motyl, 2012).”
Facts Are More Important Than Novelty (2014, background): “…(e.g., Ioannidis, 2005a; Rhine, 1934/1997; Schooler, 2011), overreliance on null hypothesis testing (e.g., Bakan, 1966; Cohen, 1994; Johnson, 2013; LeBel & Peters, 2011; Lykken, 1968; Rozeboom, 1960; Wagenmakers, Wetzels, Borsboom, & van der Maas, 2011), experimenter/researcher degrees of freedom…”
How Trustworthy Is the Scientific Literature in Industrial and Organizational Psychology? (2013, methodology): “One can divide our scientific knowledge into theory-relevant and method-relevant beliefs (LeBel & Peters, 2011).”