The press release headline for a new paper reads “Study: Low status men more likely to bully women online”, and the original article (ungated) is titled “Insights into Sexism: Male Status and Performance Moderates Female-Directed Hostile and Amicable Behaviour”.
What is the source of these deep insights into sexism? As SirT6 put it on r/science,
“Low status” in the context of this research means guys who are bad at Halo 3.
The headlines, and the introductory portions of both the popular and academic versions of the article, state findings that are much more forceful and general than the actual article. (Incidentally the press release is also by the original authors, so they are responsible for what it says.) The core finding of the study is that the number of positive comments made by male players towards female players goes up as the male players skill rises. They find nothing for negative comments. It’s pretty hard to sell.
I find the paper itself pretty interesting, and I do think the results fit the authors’ hypotheses. I do take issue with their inclusion of players’ in-game kills and deaths as predictors in their regression: those are outcome variables, and putting them in your regression on the right-hand side will lead to biased coefficient estimates. I also think the graphs overstate their findings and should show the binned values of the actual datapoints, and the confidence intervals for their fitted curves. But it’s a nice contribution in a clever setting and it has value.
What I see as a much bigger problem with the paper is how the research is being pitched, and it’s really not the authors’ fault at all. A lot of media attention goes toward the systematic biases in scientific research due to publication bias and selective data analysis choices, but there is a huge elephant in the room: framing and salesmanship. The success of a paper often depends more on how the authors frame it (relative to the audience of the journal, their specific referees, and the wider public) than on the fundamental scientific value of the work. In part, this is another way of saying that successful scientists need to be good writers.
But it also means that there is an incentive to oversell one’s work. I see this quite a bit in economics, where papers that are fairly limited in their implications are spun in a way that suggests they are incredibly general. All the incentives point in this direction: it’s hard to make it into a top general-interest journal without claiming to say something that is generally interesting.
The spin here is particularly strong: the paper that shows that having a female teammate causes male Halo 3 players to make more positive comments if they are high-skilled and fewer if they are low skilled, and got sold as saying that males online in general engage in more bullying if they are low-status. It’s often more subtle, leaving even relatively knowledgeable readers clueless about the limitations of a work, and leading to conclusions about a topic that are unsupported by actual evidence.