The Problems in Using Fixed-Effects Models of Meta-Analysis on Real-World Data

By Andy Field. 2003

242 citations recorded by the source corpus.

There are 2 approaches to meta-analysis: One assumes that studies in a meta-analysis are sampled from populations with the same effect size (the fixed-effects case), the other assumes that studies are taken from populations that have varying effect sizes (the random-effects case). As such, 2 corresponding meta-analytic frameworks have been developed: fixed- and random-effects methods. Recent evidence suggests that the assumption of fixed population effect sizes is not tenable for virtually all real-world data (e.g., Hunter & Schmidt, 2000), and yet fixed-effects methods of meta-analysis are routinely applied to real-world data (see National Research Council, 1992). This article describes some of the problems in using fixed-effects models on random-effects data by presenting 2 Monte Carlo simulations. In keeping with statistical theory (e.g., Hunter & Schmidt, 2000) results show a radical inflation of the significance tests of the mean effect sizes (above and beyond theoretical predictions). These results ...

Subfields: Applied Psychology, Management Science and Operations Research, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/9fe361fee7f5daec25a2a3c80bbf0cc177736500

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Applied Psychology

In Management Science and Operations Research

In Statistics and Probability

Citations in the corpus

How to do a meta-analysis. (2010, methodology): “On the former point, many writers have argued that real-world data in the social sciences are likely to have variable population parameters (Field, 2003b; Hunter & Schmidt, 2000, 2004; National Research Council, 1992; Osburn & Callender, 1992).”
Assessing heterogeneity in meta-analysis: Q statistic or I2 index? (2006, background): “The between-studies variability is due to the influence of an indeterminate number of characteristics that vary among the studies, such as those related to the characteristics of the samples, variations in the treatment, variations in the design quality, and so on (Brockwell & Gordon, 2001; Erez, Bl”
Motivational climate interventions in physical education: A meta-analysis (2011, methodology): “A random effects model 3 (Hedges & Olkin, 1985; Hedges & Vevea, 1998; Field, 2003) was selected for the analysis due 4 to variation between intervention methods, potential sampling error, and the possibility of 5 random unexplained variance between studies.”
Is actual similarity necessary for attraction? A meta-analysis of actual and perceived similarity (2008, background): “Monte Carlo simulations, for example, indicate that the Type I error rate (which is usually set at .05) ranges between .43 and .80 in heterogeneous fixed-effects models (Field, 2003).”
A meta-analysis of risk factors for post-traumatic stress disorder in children and adolescents. (2012, methodology): “First, real-world data are likely to have variable population parameters (Field, 2003; Hunter & Schmidt, 1990, 2000; National Research Council, 1992; Osburn & Callender, 1992).”
Revisiting the affect regulation model of binge eating: a meta-analysis of studies using ecological momentary assessment. (2011, methodology): “A randomeffects model was more appropriate than a fixed-effects model because we expected true population ES to vary across studies due to differences in sample and methodology commonly associated with “real-world” data (Field, 2003).”
Assumptions in research in sport and exercise psychology (2009, background): “As authors of meta-analyses will attest, conducting such analyses can be quite problematic as often studies report insufficient effect size data, or even lack sufficient data to calculate or infer an effect size, which means the study must be excluded from the analysis unless the author can be conta”
Ego depletion and the strength model of self-control: a meta-analysis. (2010, background): “…to correct effect sizes for sampling error variability.2 Monte Carlo simulation studies have supported the adoption of a random effects model because it permits the generalization of corrected effect sizes to the population (Field, 2003; Hunter & Schmidt, 2000; Kisamore & Brannick, 2008).”
Emotion-regulation strategies across psychopathology: A meta-analytic review. (2010, methodology): “In order to model these unconditional inferences, we ran random-effect models, as they assume that effect sizes differ from the population by sampling error plus random variability among the studies (Field, 2003; Lipsey & Wilson, 2001; Rosenthal & DiMatteo, 2001).”
Bilingualism and working memory capacity: A comprehensive meta-analysis (2017, methodology): “Furthermore, applying a fixed-effects model to the population can inflate type-I error rates from the usual 5% to anywhere from 11%–80% (Field, 2003; Hunter and Schmidt, 2000).”
The Dyadic Adjustment Scale: A Reliability Generalization Meta-Analysis (2006, methodology): “Additionally, fixed-effects models tend to produce results with inflated Type I errors when the population effect size variances are heterogeneous (Field, 2003a).”
Activity Achievement Emotions and Academic Performance: A Meta-analysis (2021, background): “Results showed no evidence of moderation for enjoyment (β = −.009, SE = .005, [95% CI = −.018, .001]), boredom (β = .003, SE = .004, [95% CI = −.005, .010]), or anger (β = .006, SE = .008, [95% CI = −.010, .022]), with all CIs encompassing zero.”
Commitment across cultures: A meta-analytical approach (2009, background): “A more detailed discussion of the different approaches can be found in Field (2003), Hedges and Olkin (1985), Hox and de Leeuw (2003), Konstantopoulos and Hedges (2004), and van den Noortgate and Onghena (2001).”