Chris Blattman has another post – his most pointed and strongest yet – telling people to get out of field experiments because the market is crowded:
Most field experiments have the hallmarks of a bad field research project. There are four:
- Takes a long time. Anything requiring a panel survey a year or two apart, or a year of setup time, suffers from this problem.
- Risky. There are a hundred reasons why any ambitious project may fail, and many do.
- Expensive. This is driven by any kind of primary data collection, but especially panel or tracking surveys, and especially any Africa or conflict/post-conflict research.
- High exit costs. This is where experiments excel. If your historical data collection, formal theory, or secondary dataset isn’t working for you, you can put it aside. If your field experiment goes poorly, not only are you stuck with it to the bitter end, but it will take more not less time.
These are all important considerations for any research project, but I was more struck by his aside that he is “suspicious whenever someone puts the method before their question.” Do people running field experiments put the method first? I would argue that they do so less than folks who write (credible) non-experimental social science papers.
The procedure for writing a paper based on a field experiment is 1) think of something you’d like to study and 2) try to come up with an experiment that lets you study it. What about non-experimental papers? Academic lore holds that the current process for grad students writing in economics is 1) sit in a room for four years trying to think of a natural experiment* that happened somewhere and 2) write a paper about whatever that natural experiment is. This is why, for example, we know a lot about the financial returns to education for students who would drop out of school if not for rules that force them to stay until age 17, or the benefits of getting a GED for someone who barely passes the necessary exam.
Let’s take a concrete example: the price elasticity of labor supply. Should we care about the labor supply of cab drivers? Trick question – it doesn’t matter whether taxi driver labor supply is interesting! What’s important is that variations in weather mean that their effective wage changes at random, so we can study their labor supply. That’s where the light is.
In contrast, experiments let us pick our topic and then study it. For example, Jessica Goldberg ran an experiment studying the exact same issue (how labor supply responds to changes in wages) but with a representative sample of Malawians, doing the most common kind of paid work in the country (informal agricultural labor). This kind of work is also common in much of sub-Saharan Africa. Her method – a field experiment – let her pick the topic of her research, and as a result what she studied is the most important category of labor across a wide region.
I’m not saying that Camerer et al.’s cab driver paper isn’t good research, or even that Goldberg’s paper is better. My claim is much simpler, and very hard to dispute: the former paper’s topic was much more driven by its method (finding a useful instrument) than was the latter’s by its method of setting up a targeted experiment.
There are exceptions to this pattern – sometimes a government agency or an NGO has an experiment they want to run that falls into your lap, for example, and some IV-driven research is based on a targeted search. In general, however, it’s misleading to claim that the experimental method often comes before the topic. That’s a key advantage of running experiments: an RCT lets us choose where to shine the light instead of constantly standing under streetlamps.
I suspect this isn’t what Blattman was driving at – there are topics where observational research is more appropriate (or even the only option, e.g. almost anything in international trade) and we shouldn’t stop studying them just because we can’t do RCTs on them. Nevertheless, the knee-jerk assumption that RCTs are methods-driven rather than topic-driven is pretty common, and, I think, wholly misguided.