I had a conversation about designing experiments a while ago that I thought was worth putting up here, in case some other stressed-out development economist Googles this same issue. A friend of mine, who I’ll leave anonymous, was worried that people would find something to critique about their experimental design. Rather than using a random number generator, their design assigned respondents to different experimental arms using an alternating pattern. Is this okay? The worry is that someone might notice it during a seminar and use it to make them look bad.
The answer is that it’s absolutely fine unless someone has rigged the order of your respondents in order to try to thwart you. Formally, let’s suppose there are two treatments, so T is equal to 1 or 0, and the outcome variable is income, I. You want to run regressions of the form I = a + bT + e, where e is the so-called omitted variable that captures all other reasons I varies besides T. You will consistently estimate b unless T is correlated with e. You’re alternating between T = 1 and T = 0, so you’ll only get into trouble if for some reason your respondents are sorted in such a way that they themselves alternate between those that are ceteris paribus rich and poor. Note that clusters of rich and poor people are fine – it’s only an issue if a large portion of the stack of surveys or whatever are exactly flipping back and forth between rich and poor. Real life doesn’t work that way – if you collected data in an actual village, it would have streaks of rich and poor households but not precise alternation.
We can go a step further: a lot of experiments don’t control T directly, but rather affect some other variable V that in turn drives T (which we tend to call an instrument or instrumental variable). The key assumption for using V to study the effect of T on I is that V is “excludable” – that it affects I only through its impact on T. Equivalently, V must be uncorrelated with e. If V itself is binary, and assigned in an alternating pattern, it’s easy to see that we’re fine – there’s no reason to think that the unobserved factors that influence I would be alternating in a fixed pattern.
But why take the word of some guy with a blog? An alternate source is the methodology section of arguably my favorite paper, Kremer and Miguel’s “Worms”, which helped kick off the current RCT-centric paradigm in development economics. Ironically, although it launched a thousand randomization ships, the worms study does not itself employ randomization:
The schools were stratified by administrative subunit (zone) and by their involvement in other nongovernmental assistance programs, and were then listed alphabetically and every third
school was assigned to a given project group.
There is no serious question as to whether this approach is valid; any issues would imply that there is some other reason why every third school, alphabetically, experienced a decline in intestinal helminths and a rise in school attendance. The goal of experiments is to achieve excludability (or exogeneity in ITT specifications), not randomization per se. We often use random assignment to achieve that goal – but that’s not the only way to do it, and other approaches can work just fine.
This is great. Thank you.