In Pursuit of Balance: Randomization in Practice in Development Field Experiments

By Miriam Bruhn, David J. McKenzie. 2008

970 citations recorded by the source corpus.

Randomized experiments are increasingly used in development economics, with researchers now facing the question of not just whether to randomize, but how to do so. Pure random assignment guarantees that the treatment and control groups will have identical characteristics on average, but in any particular random allocation, the two groups will differ along some dimensions. Methods used to pursue greater balance include stratification, pair-wise matching, and re-randomization. This paper presents new evidence on the randomization methods used in existing randomized experiments, and carries out simulations in order to provide guidance for researchers. Three main results emerge. First, many researchers are not controlling for the method of randomization in their analysis. The authors show this leads to tests with incorrect size, and can result in lower power than if a pure random draw was used. Second, they find that in samples of 300 or more, the different randomization methods perform similarly in terms of achieving balance on many future outcomes of interest. However, for very persistent outcome variables and in smaller sample sizes, pair-wise matching and stratification perform best. Third, the analysis suggests that on balance the re-randomization methods common in practice are less desirable than other methods, such as matching.

Subfields: Education, Sociology and Political Science, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Education

In Sociology and Political Science

In Statistics and Probability

Citations in the corpus

The Econometrics of Randomized Experiments (2016, background): “One paper that raised the question forcefully in this literature is Bruhn and McKenzie (2007). Theoretical paper discussing some of the formal aspects are Morgan and Rubin (2012) and Banerjee, Snowberg, and Chassang (2016). Here we offer some comments.”
Entrepreneurship Education and Entry into Self-Employment Among University Graduates (2016, methodology): “We include a binary variable for each randomization strata to increase power (Bruhn & McKenzie, 2009).”
Why Experimenters Might Not Always Want to Randomize, and What They Could Do Instead (2016, methodology): “A good general introduction to the theory of experimental design can be found in Cox and Reid (2000); a formal treatment of the theory of optimal design is given by Shah and Sinha (1989). The fact that deterministic designs might be optimal in the presence of covariates was noted by Atkinson (1982) ”
Stimulating managerial capital in emerging markets: the impact of business training for young entrepreneurs (2013, methodology): “Following the suggestions in Bruhn and McKenzie (2009), we control for strata dummies and also for baseline outcome levels in our regression analysis.”
A First Step Up the Energy Ladder? Low Cost Solar Kits and Household's Welfare in Rural Rwanda (2016, positive): “We then randomized the treatment within the 48 strata resulting from this stratification, which ensures that the resulting treatment and control groups are balanced with regards to the stratification criteria (see Bruhn and McKenzie 2009).”
Commitments to save: a field experiment in rural Malawi (2011, result): “At any rate, and to alleviate any concern that baseline imbalance may be driving our results, we follow Bruhn and McKenzie (2009) and include the full set of baseline characteristics in Table 3 as controls in our main regressions, in addition to the stratification cell fixed effects.”
Institutional Corruption and Election Fraud: Evidence from a Field Experiment in Afghanistan† (2015, background): “34 Bruhn and McKenzie ( 2009 ) suggest stratifying treatment assignment on baseline measurements of key outcomes to increase power. measures Because of fraud are unavailable prior to the election, we select our stratifying variables by identifying measures most highly correlated with fraud during…”
Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming (2023, positive): “Following the Introduc-tion Phase using Scratch, participants were divided into two groups using a matched-groups design [13, 60].”
Factorial Designs, Model Selection, and (Incorrect) Inference in Randomized Experiments (2019, background): “Bruhn & McKenzie (2009), List et al. (2011), and Athey & Imbens (2017) provide guidance on the design of field experiments, but do not discuss when and when not to implement factorial designs.”
Inference under covariate-adaptive randomization with multiple treatments (2018, methodology): “(1999), the use of this test has been recommended by Bruhn and McKenzie (2009). More recently, Bugni et al.”
Impact of SMS-Based Agricultural Information on Indian Farmers (2012, methodology): “We construct a variable b Si(Xi X) and we estimate (5) using b Si and b Si(Xi X) as instruments.”