I am currently reviewing a paper where the authors filed a lengthy analysis plan that does not actually say how they plan to analyze their data. This is an endemic problem in development economics RCTs. As a discipline we have adopted the ritual of filing pre-analysis plans (and the associated hassle cost) but not the practice of genuinely pre-specifying how we are going to look at the data.
On many of my projects, I have had coauthors insist on writing lengthy “pre-analysis plans” that are essentially papers with no data. We then duly file these on the AEA trial registry. Doing these plans is a ton of work: we spend pages and pages describing our experiments and writing about our scientific hypotheses. We cite the literature. We carefully frame our arguments. This has happened so many times, with so many collaborators, that I am not calling out any specific person here—my coauthors are simply following the norms in our field.
Those norms are wrong. None of that busywork actually pins down the specific data analyses we will do or constrains which hypothesis tests we will run. The portion of the analysis plan that is an actual plan for analyzing the data typically runs to 1-2 pages in lenght. It might be longer if the plan is super detailed about how variables will be pre-processed and cleaned. But most of the long-winded “PAPs” that typify our field do not specify any of that.
Including the extra detail in the pre-analysis plan actually makes it harder to read. As a reviewer, having to dig through a 50-page PDF to see what regressions people said they would run is a huge pain. I have taken to using ChatGPT 5.4 Thinking to find where the list of control variables is hidden (it is great at this sort of thing, by the way).
The welfare benefits of having pre-analysis plans in economics are dubious at best. We still publish many papers using secondary data where pre-specified analyses are typically impossible, and certainly not normative. I think it is better to let analyses of RCTs play with the specification a little bit to get significance stars than to skew the published literature toward less-credible research designs. Publication bias is probably smaller than not-right-in-the-first-place bias.
However, if we are going to have analysis plans then we need to focus them on the actual analysis plan part. That is my plea to development economists: stop with the writing of the contentless papers! Give me your plan for analyzing the data—and nothing else.