Beyond Baseline and Follow-Up: The Case for More T in Experiments

By David J. McKenzie. 2011

864 citations recorded by the source corpus.

The vast majority of randomized experiments in economics rely on a single baseline and single follow-up survey. If multiple follow-ups are conducted, the reason is typically to examine the trajectory of impact effects, so that in effect only one follow-up round is being used to estimate each treatment effect of interest. While such a design is suitable for study of highly autocorrelated and relatively precisely measured outcomes in the health and education domains, this paper makes the case that it is unlikely to be optimal for measuring noisy and relatively less autocorrelated outcomes such as business profits, household incomes and expenditures, and episodic health outcomes. Taking multiple measurements of such outcomes at relatively short intervals allows the researcher to average out noise, increasing power. When the outcomes have low autocorrelation, it can make sense to do no baseline at all. Moreover, the author shows how for such outcomes, more power can be achieved with multiple follow-ups than allocating the same total sample size over a single follow-up and baseline. The analysis highlights the large gains in power from ANCOVA rather than difference-in-differences when autocorrelations are low and a baseline is taken. The paper discusses the issues involved in multiple measurements, and makes recommendations for the design of experiments and related non-experimental impact evaluations.

Subfields: Economics and Econometrics, Gender Studies, Sociology and Political Science

View Paper: https://www.semanticscholar.org/paper/d6ef815b332cc07a3169c6bcabc16bc963d58a5f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Economics and Econometrics

In Gender Studies

In Sociology and Political Science

Citations in the corpus

Seeing is Believing? Evidence from an Extension Network Experiment (2014, positive): “We therefore collected two follow-up surveys to improve our ability to detect a signi cant impact, following McKenzie (2012).”
The Impact of Cash and Food Transfers: Evidence from a Randomized Intervention in Niger (2018, neutral): “Impact evaluations usually have baseline surveys prior to the start of the intervention, though as McKenzie (2012) notes, this is not always necessary.”
The impact of microcredit on agricultural technology adoption and productivity: Evidence from randomized control trial in Tanzania (2020, methodology): “Following Frison and Pocock (1992) and McKenzie (2012), we also estimate the Analysis of Covariance (ANCOVA) regression for the robustness check, by including the outcome variable at the baseline as the control variable in estimating both ITT and LATE.”
The Effect of Cash, Vouchers, and Food Transfers on Intimate Partner Violence: Evidence from a Randomized Experiment in Northern Ecuador (2016, neutral): “Moreover, we estimate the treatment effect using Analysis of Covariance ( ANCOVA ) , which 2012 .”
Cash, food, or vouchers? Evidence from a randomized experiment in northern Ecuador (2014, neutral): “Given the high variability and low autocorrelation of our food consumption data, ANCOVA estimates are preferred over difference-in-difference estimates (McKenzie, 2012).”
Financial Literacy and Financial Behaviour: Experimental Evidence from Rural Rwanda (2016, methodology): “…ordinary least square (OLS) models, controlling for a vector of covariates, as follows: where Y it denotes an outcome variable for respondent i (in 2012), D i is a dummy variable equal to 1 if the respondent received training, X ij is a vector of j covariates measured at the baseline (in 2011),…”
Returns to microcredit, cash grants and training for male and female microentrepreneurs in Uganda (2018, methodology): “The useofmultiple data collections helps to alleviate this issue, as discussed in McKenzie (2012). Stratificationwas done by region but not for any other characteristic due to program implementation beginning before the baseline data was fully analyzed.”
Exporting and Firm Performance: Evidence from a Randomized Experiment (2017, neutral): “MCKENZIE, AND J. ROBERTS (2013): “Does Management Matter? Evidence from India,” The Quarterly Journal of Economics, 128, 1–51.”