Why do we ever believe in scientific evidence?

Late last night the internet exploded with the revelation that the most influential social science study of 2014 was apparently invented from whole cloth. LaCour and Green (2014) showed that a brief interaction with an openly-gay canvasser was enough to change people’s minds – turning them from opposing gay rights to supporting them. The effects were persistent, and massive in magnitude. This study was a huge deal. It was published in Science. It was featured on This American Life. I told my non-academic friends about it. And, according to an investigation by Broockman, Kalla, and Aronow, as well as a formal retraction by one of the study’s authors, it was all made up. The report by Broockman, Kalla, and Aronow is a compelling and easy read – I strongly recommend it.

The allegation is that Michael LaCour fabricated all of the data in the paper by drawing a random sample from an existing dataset and adding normally-distributed errors to it to generate followup data. I have nothing to add to the question of whether the allegation is true, other than to note that many people are persuaded by the evidence, including Andrew Gelman, Science, and LaCour’s own coauthor, Donald Green.

What I do have to add is some thoughts on why I trust scientists. Laypeople often think that “peer review” means the kind of analysis that Broockman, Kalla, and Aronow did – poring over the data, looking for mistakes and fraud. That isn’t how it works. Referees are unpaid, uncredited volunteers who don’t have time  to look at the raw data themselves. (I have also never been given the raw data to look at when reviewing an article). The scientific method is fundamentally based on trust – we trust that other scientists aren’t outright frauds.* Nothing would work otherwise.

Why do we trust each other? After all, incidents like this one are not unheard of. Speaking from my own experience running field experiments, one important reason is that faking an entire study would be really hard. You’d have got to write grants, or pretend you’re writing grants. Clear your study with the IRB, or sometimes a couple of IRBs. You’d have to spend a significant portion of your life managing data collection that isn’t really happening, and build a huge web of lies around that. And then people are going to want to see what’s up. This American Life reports that LaCour was showing his results to the canvassers he worked with, while the data was coming in (or supposedly coming in). To convincingly pull all of this off, you would basically have to run the entire study, only not collect any actual data.

It is hard to imagine anyone who is realistic with themselves about the incentives they face choosing to go through with all of this. Most scientists don’t get hired by Princeton, don’t make tons of money, don’t get their results blasted across all media for weeks. Most of them work really hard for small payoffs that seem inscrutable to outsiders. The only way to get big payoffs is with huge, sexy results – but huge results invite scrutiny. You might get away with faking small effects that are relatively expected, but if your study gets attention, people are going to start digging into your data. They will try to replicate your findings, and when they can’t they will ask questions. If you do manage to walk the tightrope of faking results and not getting caught, you did a ton of work for nothing.

I can barely conceive of going through all the effort and stress of running a field experiment only to throw all that away and make up the results. I trust scientists to be honest because the only good reason to go into science is because you love doing science, and I think that trust is well-placed.

*Incidentally, this is why I don’t particularly blame Green for not realizing what was up. When I coauthor papers with people, the possibility that they are just making stuff up never even crosses my mind. I am looking for mistakes, sharing ideas, and testing my own ideas and results out – not probing for lies. News accounts show that Green did see the data used for the analysis, just not the underlying dataset or surveys.

1 thought on “Why do we ever believe in scientific evidence?”

  1. Collect data yourselves, you naive idiots. There is a whole community of data vampires who love nothing more than “inheriting” data sets to prove their pet hypotheses. The only way to advance knowledge is to get out there and get your hands dirty, sleep in the field, see and understand the way people live.

Leave a Reply

Your email address will not be published. Required fields are marked *