Anyone who has ever talked to me about science has probably heard me get all excited about all sorts of methodologies. I am particularly likely to get all flushed by very simple methods that manage to get at something that seemed hard to get good data about. I have on several occasions on this blog mentioned various experiments that have measured not espousals of beliefs in superstitious claims, i.e. what you get with questionnaires, but actual superstitious behaviour. The problem with questionnaires being that people tend to write what they think they should instead of what it actually true (assuming they even know their mind enough to know what is true).
Just recently, however, I heard about a way of slightly altering the methodology with surveys that sounds like a really cool way of getting people to be more truthful:
Getting honest answers about behaviour that is illegal or frowned-upon – such as taking drugs or visiting prostitutes – is notoriously difficult. But survey researchers have devised a neat way to get people comfortable with revealing their indiscretions.
Each time the researcher asks the respondent a question, the respondent throws dice before answering – crucially, the researcher cannot see what numbers come up. The rules of the game will be something like this: the respondent will always answer “yes” if they throw a six and “no” when a one comes up, but should tell the truth otherwise.
Because a “yes” doesn’t necessarily mean that the respondent actually committed the undesirable behaviour, people seem to open up. The forced “yes” and “no” answers introduce some “noise” into the results, but overall this “randomised response technique” (RRT) gives better answers. For instance, RRT questions get much closer than conventional surveys to the actual incidence of drug use that is revealed by screening tests on hair samples.
I wonder to what degree this kind of methodology would be useful when dealing with supernatural beliefs given that belief in superstitions is often looked down upon and therefore surveys concerning it seem like a natural place to use RRT. I had seen many such surveys and never heard of a single one that used this method, which can be explained by two hypotheses: either the researchers had not heard of this trick or they did not believe that it would be useful. The first of those alternatives is kind of boring and weird given that wikipedia informs us that the method has been around since 1965, the second much less so as it would have to hinge on an understanding of these surveys that I do not yet have. Any suggestions?