Conclusions beyond support: overconfident estimates in mixed models

By H. Schielzeth, Wolfgang Forstmeier. Behavioral Ecology, 2008

899 citations recorded by the source corpus.

Mixed-effect models are frequently used to control for the nonindependence of data points, for example, when repeated measures from the same individuals are available. The aim of these models is often to estimate fixed effects and to test their significance. This is usually done by including random intercepts, that is, intercepts that are allowed to vary between individuals. The widespread belief is that this controls for all types of pseudoreplication within individuals. Here we show that this is not the case, if the aim is to estimate effects that vary within individuals and individuals differ in their response to these effects. In these cases, random intercept models give overconfident estimates leading to conclusions that are not supported by the data. By allowing individuals to differ in the slopes of their responses, it is possible to account for the nonindependence of data points that pseudoreplicate slope information. Such random slope models give appropriate standard errors and are easily implemented in standard statistical software. Because random slope models are not always used where they are essential, we suspect that many published findings have too narrow confidence intervals and a substantially inflated type I error rate. Besides reducing type I errors, random slope models have the potential to reduce residual variance by accounting for between-individual variation in slopes, which makes it easier to detect treatment effects that are applied between individuals, hence reducing type II errors as well.

Subfields: Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/099aaf6fa953b77cc04ea8d89af1edb4dcd3ec52

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistics and Probability

Citations in the corpus

Linear models and linear mixed effects models in R with linguistic applications (2013, positive): “ Moreover, researchers in ecology (Schielzeth & Forstmeier, 2009), psycholinguistics (Barr, Levy, Scheepers, & Tilly, 2013) and other fields have shown via simulations that mixed models without random slopes are anti-‐ conservative or,…”
Diving seabirds share foraging space and time within and among species (2010, methodology): “…potential individual differences with regard to preferred distances to the coast, daytimes for foraging trips as well as cube and bathymetric depths we also included random slopes for these effects (and their squares except for daytime) into the model (Baayen 2008, Schielzeth and Forstmeier 2009).”
Quantifying individual variation in behaviour: mixed-effect modelling approaches. (2013, background): “Despite major advantages, MMs are complex tools and therefore easily misspecified or interpreted inappropriately (Bennington & Thayne 1994; Bolker et al. 2009; van de Pol & Wright 2009; Schielzeth & Forstmeier 2009; Zuur et al. 2009; Hadfield et al. 2010).”
Model selection and model averaging in behavioural ecology: the utility of the IT-AIC framework (2010, background): “In this case, treating all data as independent may result in incorrect inference regarding treatment effects (Schielzeth and Forstmeier 2009).”
A brief introduction to mixed effects modelling and multi-model inference in ecology (2018, background): “Here, ‘body mass’ is specified as a random slope by moving it into the random effects structure: M4 <- glmer(successful.breed ~ body.mass + (body.mass|sample.site) Schielzeth & Forstmeier (2009) warn that constraining groups to share a common slope can inflate Type I and Type II errors.”
Multimodel inference in ecology and evolution: challenges and solutions (2011, background): “Using both random intercepts and slopes reduces the incidence of Type I and Type II errors and reduces the chance of overconfident estimates (unrealistically low standard error, SE) (Schielzeth & Forstmeier, 2009).”
Two‐current choice flumes for testing avoidance and preference in aquatic animals (2017, background): “In some such cases, random slope models may be useful (Schielzeth & Forstmeier 2009), and although the application of random slopes may account for within-individual pseudoreplication with respect to the covariates to which they are applied (e.g. trial time), themain purpose of using random slopes…”
The Utility and Application of Mixed-Effects Models in Second Language Research (2015, background): “…stress the importance of random slopes in a repeated measures design, as not including random slopes in the presence of considerable random slope variance can lead to drastically increased Type I error rates (for further discussion, see Barr 2013; Barr et al. 2013; Schielzeth & Forstmeier, 2009).”
Meta‐analysis of variation: ecological and evolutionary applications and beyond (2015, neutral): “An equivalent model to Equations (15–17) can be written as (cf. Schielzeth & Forstmeier 2009): ln r̂j ¼ ðb0 þ siÞ þ ðb1 þ uiÞGroupj þ b2 ln xj þ ej þmj; eqn 18 si ui N 0 0 ; r2s qrsru qrsru r2u ; eqn 19 ej Nð0;r2eÞ; eqn 20 mj Nð0;r2lnrjÞ; eqn 21 where ln r̂j is the jth effect size as in Equation…”
Balancing Type I Error and Power in Linear Mixed Models (2015, background): “Most importantly, by fitting the maximal model, one avoids the risk of falsely removing a variance component which may lead to anti-conservative test statistics for fixed effects (e.g., Barr et al., 2013; Schielzeth & Forstmeier, 2009).”
A general and simple method for obtaining R2 from generalized linear mixed‐effects models (2013, background): “Snijders & Bosker (1999) point out that calculating R2 like R21 and R 2 2, it is easy to do so for random intercept models, but for random-slope models is tedious (as variance components for slopes cannot be easily integrated with other variance components, e.g. Schielzeth & Forstmeier 2009).”