Any outsider who’s visited rural Africa knows that outsiders, especially white folks, are fairly rare in those parts. As I’ve written before, I’m the object of curious stares and the subject of ridiculous rumors whenever I venture out into the set of villages that comprise my catchment area. It’s no surprise, then, that my presence might have an effect on how people respond to my survey.
Data is not the plural of anecdote, however – we need an actual, rigorous study to know that the presence of white folks matters, and especially to measure how exactly people’s behavior changes. What do they do differently? By how much? Jacobus Cilliers of CSAE describes exactly what is needed to answer those questions – along with Oeindrila Dube and Bilal Siddiqi he randomly assigned either a white person or a black person to sit in on an interview. The black person was in fact Sierra Leonese (the experiment was in Sierra Leone), but this experimental person didn’t talk, so that was really up to the subjects to infer from the person’s appearance – as was the presumable guess that the white person was a foreigner and probably British. In the interviews the subjects played the ever-popular dictator game, where one person is given a certain amount of cash and allowed to decide what portion to give to another person. They found that Sierra Leonese people are more generous in the presence of a white person than when all the researchers are black – to the tune of a 20% increase in giving to the other player in the games.
This is a cool idea, and really valuable research – we need to measure these kinds of effects so that we can assess the external validity of our research. Just because we’ve done a valid study for the population and setting chosen doesn’t mean that the results will carry over to the case where, for example, the white NGO representatives aren’t around. But I disagree with some of Jacobus’s advice on what to do to mitigate these outsider effects:
First, your presence in the research significantly undermines the results. You should either be absent when games and surveys are conducted, or randomly vary your presence. The success of impact evaluations could thus be overstated, if respondents “give the right answers” to please the researcher. Moreover, true behavioural change could be short-lived if it is determined by the presence of a “white man”, rather than the actual project.
Let’s start with the last part – our the results of impact evaluations overstated if, say, a white guy, is, for example, sitting in the project minibus writing blog posts while the surveys and interventions are going on? No. Remember that a valid impact evaluation involves randomization – participants are assigned to the treatment or control group strictly by chance, and therefore in expectation differ only in terms of whether they receive the treatment. The results of impact evaluations would be overstated only in the case where outsider effects *only* apply to members of the treatment group. There’s no reason to think that should be true.
Indeed, outsider effects are more likely to attenuate the measured results of an impact evaluation. Suppose that my being around makes people want to appear to be good citizens, and to underreport risky sex – that is, to lie. In the extreme, everyone reports no risky sex at all. If I use self-reported risky sex as my outcome, then I would find no effect of my intervention at all, even if it’s actually working.
What about the first point – does our presence significantly undermine the results of our research? The best we can say is “maybe”. The Cilliers et al. paper is a great first step, but the underlying research project was the dictator game, which is not itself an experiment. If you want to know if impact evaluation results are undermined by an outsider’s presence, you need to do a real RCT of some kind with a treatment and control group, and then do a cross-cutting randomization of whether outsiders are present. This is effectively two RCTs, which can be validly compared – one with with folks around and one without. Then you could see whether the measured effects differ between the variants of the study. Inasmuch as our presence does undermine the results, though, the effect is not to overstate the impact of a given intervention, but to understate it. So maybe we don’t care – lower bounds are very useful for discussing cost-effectiveness.
Now, what can we do about this? I totally agree with one of Cilliers’s suggestions – the best data is likely to come from not being around at all. The other option – to randomly vary one’s presence – is a very bad suggestion. First off, it’s not necessary – if the white guy is around for every village in the sample, then he has no differential effect across the treatment and control groups, and the results remain internally valid. Second, because the white guy’s presence has an effect on the results, randomly varying his presence may decrease the precision of any estimated effects. If the white guy’s presence is random, it will have on average the same effect for the treatment and control groups, so the experiment will remain valid. However, this additional random variation cpuld increase the residual variance of the outcome measure, which is bad for precision. Imagine that your outcome is unprotected sex, and everyone actually has unprotected sex 7 times a week if left to their own devices. The treatment causes them to decrease this to 4 times a week. However, the effect of the azungu being around is to cause everyone to shade down their risk-taking by 3 times per week – reporting 4 times and 1 time respectively. Then you estimate the same treatment effect whether the white guy is always around, never around, or around at random – it doesn’t matter if he’s there, as long as his presence is uncorrelated with the treatment. But the standard errors of the estimated effects are much larger if his presence varies randomly.
To see why, notice that we could replace the white guy actually showing up with a coin flip that randomly decreases people’s answers by 3. “Whitey never around” is a coin with tails on both sides – no answers change. “Whitey always around” is a coin that’s heads on both sides – all the answers change. This is fine! I could take your experimental data and add/subtract any number to all the observations, and the results would be unchanged; the difference between the treatment and control means would be the same. What about the “whitey randomly around” case? That’s a regular coins, one heads side and one tails side. I am literally adding random noise to the experimental data. That’s going to make my measurements less precise, and decrease my statistical power. Now, we can control for this since we know where whitey showed up – but even if we do, it gains us little. It’s also another thing to explain in our analysis. Randomizing whether the white guy is around is not a great idea – unless the goal is to measure the effect of the white guy on the results.
More broadly, let’s not forget that white guy* effects are only one variant of outsider effect. Irrespective of my presence, any survey project in rural africa necessarily involves the arrival of the following novelties in a village: 1) a car – kids love staring at and playing with it; 2) a bunch of folks from other villages, all at once and really well-dressed; 3) a bunch of compensation payments; and 4) people asking about all kinds of weird stuff like sex, whether you can read, and your income. Nothing we can do will eliminate any possible influence this stuff has on our results.
Hat tip: Development Impact
Jason – Why are you assuming that the effect of the presence of the outsider researcher would be independent of the effect of the treatment? Your example of the reports of 7&4 vs. 4&1 assumes a zero interaction. But I think it’s easy to imagine cases where the presence of the investigator (as opposed to a local, trained to make a similar intervention) might change the treatment effect. For example, we can imagine that people are more (or less) likely to use a free/reduced price bednet if they see the outside researcher wandering around – but those in villages where the price was not discounted might not change their behavior (since there is no implied gift relationship or whatever). Isn’t that at least plausible?
That’s totally credible, and is a strong argument for people to do some RCTs with cross-cutting white guy randomizations to get a sense of where this matters. Until we have those, we need to use whatever other evidence we have (common sense? talking to people?) to infer when it matters.
But it also raises the question of what intervention we are testing. Is it something that will be implemented by locals, with no white folks around? Then keep white people away from all the treatment villages. Is it something that will be implemented partly by the white people in question? Then they should be around, playing the role that they would on the actual implemented project. I’d say in that case you also want them around for the control group, to account for whatever Hawthorne effects are created just by having the white people around.