Rigorous Large-Scale Educational RCTs Are Often Uninformative: Should We Be Concerned?

By Hugues Lortie-Forgues, Matthew Inglis. Educational Research, 2019

154 citations recorded by the source corpus.

There are a growing number of large-scale educational randomized controlled trials (RCTs). Considering their expense, it is important to reflect on the effectiveness of this approach. We assessed the magnitude and precision of effects found in those large-scale RCTs commissioned by the UK-based Education Endowment Foundation and the U.S.-based National Center for Educational Evaluation and Regional Assistance, which evaluated interventions aimed at improving academic achievement in K–12 (141 RCTs; 1,222,024 students). The mean effect size was 0.06 standard deviations. These sat within relatively large confidence intervals (mean width = 0.30 SDs), which meant that the results were often uninformative (the median Bayes factor was 0.56). We argue that our field needs, as a priority, to understand why educational RCTs often find small and uninformative effects.

Subfields: General Health Professions, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/a64eaee094b3acda990061d64ac7680901d02b04

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Building Content Knowledge to Boost Comprehension in the Primary Grades (2020, unknown): “Indeed, large-scale randomized controlled trials typically demonstrate an average effect size of 0.01 on kindergarten outcomes and 0.01 on language outcomes across grades (Lortie-Forgues & Inglis, 2019).”
Tackling Educational Inequalities with Social Psychology: Identities, Contexts, and Interventions (2020, background): “…or providing targeted academic or practical support (Education Endowment Foundation, 2019) have met with some success, but tend to be expensive and/or resource intensive and in many cases have only fairly modest effects (Boulay et al., 2018; Lortie-Forgues & Inglis, 2019; Yeager & Walton, 2011).”