Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling

By Leslie K. John, George Loewenstein, Drazen Prelec. Psychology Science, 2012

1,942 citations recorded by the source corpus.

Subfields: Safety Research, Social Psychology, Sociology and Political Science

View Paper: https://www.semanticscholar.org/paper/3813c9cb1bbbf699998b622fe4c8dbb02c9db482

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Safety Research

In Social Psychology

In Sociology and Political Science

Citations in the corpus

The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research* (2020, neutral): “In a hypothesis-testing study powered to detect an effect on the primary outcome measure, data on secondary outcomes are used to evaluate additional effects of the intervention, but subsequent statistical analysis of secondary outcome measures may be underpowered, making results and interpretation l”
Reporting animal research: Explanation and Elaboration for the ARRIVE guidelines 2019 (2019, background): “In a hypothesis-testing study powered to detect an effect on the primary outcome measure, data on secondary outcomes are used to evaluate additional effects of the intervention but subsequent statistical analysis of secondary outcome measures may be underpowered, making results and interpretation le”
Choosing prediction over explanation in psychology: Lessons from machine learning (2017, background): “There is increasingly broad agreement that this widespread replication failure is largely due to “ p -hacking” and other questionable research practices that have historically been prevalent in the field (John, Loewenstein, & Prelec, 2012; Simmons et al., 2011).”
A solution to the single-question crowd wisdom problem (2017, background): “To discourage such behaviour, one can impose truth-telling incentives with the Bayesian truth serum, which also elicits respondents’ vote predictions 24,25 .”
Degrees of Freedom in Planning, Running, Analyzing, and Reporting Psychological Studies: A Checklist to Avoid p-Hacking (2016, background): “…in previous publications (Kriegeskorte et al., 2009; Nieuwenhuis et al., 2011; Simmons et al., 2011; Wagenmakers et al., 2011; Bakker et al., 2012; John et al., 2012; Chambers, 2013; Francis, 2013; Bakker and Wicherts, 2014; Simonsohn et al., 2014; Steegen et al., 2016; van Aert et al., 2016).”
Scientific Utopia: II. Restructuring Incentives and Practices to Promote Truth Over Publishability (2012, background): “, 2011; Lord, Ross, & Lepper, 1979); (c) selectively reporting studies with positive results and not studies with negative results (Greenwald, 1975; John et al., 2012; Rosenthal, 1979) or selectively reporting "clean" results (Begley & Ellis, 2012; Giner-Sorolla, 2012); (d) stopping data collection ”
Three ways to make replication mainstream (2017, background): “, Bakker, M., Maassen, E., & Wicherts, J. (2018). Verify original results through reanalysis before replicating: A commentary on “Making replication mainstream” by Rolf A.”
Editorial Essay: The Tumult over Transparency: Decoupling Transparency from Replication in Establishing Trustworthy Qualitative Research* (2019, background): “…Oxford disciplines increasingly recognize that a sizable portion of published quantitative findings has been based on questionable research practices (John, Loewenstein, and Prelec, 2012), the legitimacy of the quantitative research enterprise is coming under threat (Anderson, Wennberg, and…”
Estimating the Reproducibility of Experimental Philosophy (2018, neutral): “Gettierized Knobe effects. Episteme, 10(3), 219. (Content-based, successful, osf.io/k89fc/) Random 2: Rose, D., & Nichols, S. (2013). The lesson of bypassing.”
The preregistration revolution (2018, background): “Moreover, researchers have substantial degrees of freedom to conduct many different tests, and selection of those that yield positive results over those that yield negative results will increase the likelihood of attractive results at the expense of accuracy (30, 41, 42).”
The Replication Crisis in Psychology: An Overview for Theoretical and Philosophical Psychology (2019, background): “Because researchers experience strong pressure to publish in high-impact outlets and because such outlets generally have a reputation for publishing novel and positive findings, critics suggest that these pressures can lead researchers to engage in QRPs that will make their findings more attractive ”
Racial Inequality in Psychological Research: Trends of the Past and Recommendations for the Future (2020, neutral): “…field’s recent emphasis on metasci-ence, including decreasing false-positive findings and increasing replication efforts and open-science practices (John, Loewenstein, & Prelec, 2012; Makel, Plucker, & Hegarty, 2012; Pritschet, Powell, & Horne, 2016; Sim-mons, Nelson, & Simonsohn, 2011), we…”
Facts Are More Important Than Novelty (2014, background): “This difference raises questions regarding potential biases in replicating one’s own work and may be related to previous findings of questionable research practices in the social sciences (John et al., 2012; Simmons et al., 2011).”
Is psychology suffering from a replication crisis? What does "failure to replicate" really mean? (2015, background): “Questionable research practices such as those identified by John et al. (2012) and Simmons et al. (2011) clearly need to be addressed because they produce inflated estimates of effect sizes and render p values largely uninterpretable.”