Andrew Gelman links to Casey Mulligan’s take on whether survey respondents should be compensated for their time. Mulligan suggests that from the researcher’s perspective, paying people might lead to better results. But he seems to miss the most important reason for paid surveys leading to better data quality: running an unpaid survey means that your refusal rate will be an increasing function of how much respondents value their time (and their impatience, and a few other thing). Your wonderful random sample is now no longer representative of the whole population. This is maybe okay, but you don’t actually know the selection process. Also, the value people put on their time is, in rough approximation, equal to their wages. Wages are a common outcome variable in analyses, and as I taught my undergraduate econometrics students last term, selecting your sample on the outcome variable means your estimates will be biased – in layman’s terms, you’ll get the wrong answer.
Mulligan’s piece reminds me of Chris Blattman’s post on this same topic last year. He discusses two possible problems with paying, raised by Aine McCarthy. The first is that people might pay less attention or participate for the wrong reasons; Blattman points out that it’s possible to run an experiment to study that stuff. They both seem unlikely to me, though.
But the second issue Blattman discusses is just bizarre to me – the idea that we might establish a social norm of paying people for their time (okay, plausible) and that that somehow is a bad thing (wait, really?). I’ve seen this suggestion from a number of corners, and I don’t get it. You’re taking something of value from someone (a chunk of their time, sometimes more than a whole day); a norm that says they should be compensated for that is a great idea. These aren’t stray cats that we’re thinking about feeding – they’re people, who have other stuff going on, and, as I’ll discuss, may feel obliged to participate.
One concern often raised by non-economists and mentioned in the comments on Blattman’s post is that paying people, especially if you pay them too much relative to their income, is coercive. Maybe it’s indicative of my training as an economist that I find that argument confusing. Are jobs “coercion”? Those mostly involve being paid enough to do something you dislike.
But that’s beside the point here in Malawi, and probably in lots of other places as well. Despite researchers’ best efforts to ensure respondents that our surveys are voluntary through the informed consent process, virtually no one refuses. I’ve heard that this is because some government surveys are legally mandatory. I would feel terrible imposing on someone who thinks they are required to participate and not at least try to make it worth their while. It seems very much like stealing to me, and an abuse of my status and power in the places where I do research.
Generally, I agree, but two random thoughts. First, “jobs being coercive” sounds silly on some level but has some deeper truth to it. See the recent lengthy debates between Crooked Timber and Marginal Revolution, which centered around the extent to which workers could really opt out of work, and thus, the necessity of workplace regulations that promote workers’ substantive freedoms (like going to the bathroom…). I don’t think it really applies to this context… but partly because IRBs already exist to enforce a lot of the protections you’d want to see here.
Second, how does paying survey respondents solve the problem of response rate being a decreasing function of respondents’ wages/value of the respondents’ time? I guess if you are in a situation where you can pay enough such that for all possible wages, all respondents say yes (or at least, say no with equal probability based on other factors), you’d solve that particular selection issue. But in richer countries, at least, that rate might be prohibitive for samples involving middle and upper income respondents. How much would you have to offer someone making 100$/hr and working 80 hrs/week to do your 1-2 hour survey with equal probability to someone who is making 8$/hr working part time? Etc. It’s probably better than not offering anything for selection, but it’s not a cure-all without some subsidiary assumptions/design features. Right?
I hadn’t noticed the parallel with the workplace freedom argument. I’m pretty sympathetic to the Crooked Timber side of that back-and-forth – I generally think firms have lots of bargaining power and workers very little. So maybe jobs can be coercive, in some cases. The concept of a job, the Econ Platonic ideal of employment – with perfect competition and perfect information and so forth – probably isn’t – abusive firms would quickly be driven out of the market. Of course, very little of the labor market looks like an economics 101 model of wages and employment.
But thinking about this has made me realize that being paid to take a survey in Malawi isn’t that much like a job at all. It’s a conditional windfall transfer of money, on top of what people already survive on. So there isn’t the aspect of firing and becoming indigent that goes with jobs. It’s much more like a game show. Is (or, god willing, was) Fear Factor coercive? What about The Price is Right?
As for eliminating the selection problem, I was definitely assuming you could pay enough to get everyone to participate (basically a given in rural Malawi). In the US you would run into issues – a strong argument for maintaining the ACS as mandatory, but maybe paying people some amount for their time. Like jury duty, except that the case for picking random people for surveys seems a bit stronger than that for picking random people to decide trials.
for the record, this summer we compensated our participants a whopping 200MWK. We initially planned to do 100MWK, then devaluation happened.
This makes me feel better about my compensation, which will be worth exactly the same amount. I’m planning to use gifts (probably soap) instead in order to reduce the possibility of graft, and arguably because they’re less coercive although I’m still not sure what that means.