“Positive” Results Increase Down the Hierarchy of the Sciences

By Daniele Fanelli. PLoS ONE, 2010

677 citations recorded by the source corpus.

The hypothesis of a Hierarchy of the Sciences with physical sciences at the top, social sciences at the bottom, and biological sciences in-between is nearly 200 years old. This order is intuitive and reflected in many features of academic life, but whether it reflects the “hardness” of scientific research—i.e., the extent to which research questions and results are determined by data and theories as opposed to non-cognitive factors—is controversial. This study analysed 2434 papers published in all disciplines and that declared to have tested a hypothesis. It was determined how many papers reported a “positive” (full or partial) or “negative” support for the tested hypothesis. If the hierarchy hypothesis is correct, then researchers in “softer” sciences should have fewer constraints to their conscious and unconscious biases, and therefore report more positive outcomes. Results confirmed the predictions at all levels considered: discipline, domain and methodology broadly defined. Controlling for observed differences between pure and applied disciplines, and between papers testing one or several hypotheses, the odds of reporting a positive result were around 5 times higher among papers in the disciplines of Psychology and Psychiatry and Economics and Business compared to Space Science, 2.3 times higher in the domain of social sciences compared to the physical sciences, and 3.4 times higher in studies applying behavioural and social methodologies on people compared to physical and chemical studies on non-biological material. In all comparisons, biological studies had intermediate values. These results suggest that the nature of hypotheses tested and the logical and methodological rigour employed to test them vary systematically across disciplines and fields, depending on the complexity of the subject matter and possibly other factors (e.g., a field's level of historical and/or intellectual development). On the other hand, these results support the scientific status of the social sciences against claims that they are completely subjective, by showing that, when they adopt a scientific approach to discovery, they differ from the natural sciences only by a matter of degree.

Subfields: History and Philosophy of Science, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/b90d969393454f4517736af968d9428098fe0e98

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In History and Philosophy of Science

In Statistics and Probability

Citations in the corpus

Adjusting for Publication Bias in Meta-Analysis (2016, result): “Although studies with results that are statistically significant are overrepresented in the published literature relative to those with results that are not statistically significant (Fanelli, 2010, 2012; Sterling, 1959; Sterling, Rosenbaum, & Weinkam, 1995), it is clearly not the case that there ar”
Mind The Hype: A Critical Evaluation and Prescriptive Agenda for Research on Mindfulness and Meditation (2017, neutral): “…safety of treatment interventions have also embroiled the behavioral and neuropsychiatric sciences (Baker, McFall, & Shoham, 2008; Button et al., 2013; Fanelli, 2010; Ioannidis, 2005; Munafò, Stothart, & Flint, 2009; Simmons, Nelson, & Simonsohn, 2011; Yarkoni, Poldrack, Van Essen, & Wager, 2010).”
HARKing's Threat to Organizational Research: Evidence From Primary and Meta‐Analytic Sources (2016, neutral): “…acknowledged by authors and editors (e.g., Bedeian et al., 2010; Fanelli, 2009), authors’ evolving hypothesis statements within sources over time (O’Boyle et al., in press), and suspiciously low hypothesis falsification rates (e.g., Fanelli, 2010; Francis, Tanzman, & Matthews, 2014; Leung, 2011).”
Registered Reports A Method to Increase the Credibility of Published Results (2014, background): “As a consequence, replications (Makel, Plucker, & Hegarty, 2012) and negative results (Fanelli, 2010; Sterling, 1959) are rare in the published literature.”
The Chrysalis Effect (2017, background): “These pressures may lead researchers to engage in questionable research practices (QRPs) to bolster their chances of publication (Chen, 2011; Fanelli, 2010a, Kepes & McDaniel, 2013).”
Promoting Open Science to Increase the Trustworthiness of Evidence in Special Education (2018, background): “Moreover, studies with null findings are scarce in the published research base, especially in the social sciences (Fanelli, 2010; Sterling, 1959).”
How to Make More Published Research True (2014, background): “It may lead them to higher credibility, more akin to that of fields that have traditionally been more rigorous in this regard, like the physical sciences [44].”
The surprising persistence of Biglan's classification scheme (2017, background): “It has received widespread use in many analyses within higher education including teachers’ reflective practice (Kreber & Castleden, 2009); the relationship between class size and grades (Johnson, 2010) and even the proportion of positive result reported in hypothesis testing papers (Fanelli, 2010).”
Taking the U.S. Patent Office Criteria Seriously: A Quantitative Three-Criterion Creativity Definition and Its Implications (2012, background): “Perhaps the only advantage of this conceptual vagueness is that it is much easier for psychologists to confirm their hypotheses than is the case for scientists in the physical and biological sciences (Fanelli, 2010 ).”
Instead of "playing the game" it is time to change the rules: Registered Reports at AIMS Neuroscience and beyond (2014, background): “The last ten years have witnessed increasing awareness of questionable research practices (QRPs) in the life sciences [1,2], including p-hacking [3], HARKing [4], lack of replication [5], publication bias [6], low statistical power [7] and lack of data sharing ([8]; see Figure 1).”
Replication, statistical consistency, and publication bias. (2013, background): “For example, as noted by Sterling (1959), Fanelli (2010), and others, the field of psychology has a preference for publishing statistically significant findings.”
Recommendations for Increasing Replicability in Psychology † (2013, background): “In a comparison of publications in 18 empirical research areas, Fanelli (2010) found rates of confirmed hypotheses ranging from 70% (space”