Conceptual Replication

Here is the short commentary that I wrote about conceptual replication for the May issue of The Psychologist (PDF), published by the British Psychological Society (which is why they spell "behaviour" that way -- although as a Canadian, I don't really object). They have an entire section in the issue devoted to questions of replication in psychology (I'll have a bit more to say about that later), but they are all worth a read.  

There is no substitute for direct replication – if you cannot reproduce the same result using the same methods then you cannot have a cumulative science. But conceptual replication also has a very important role to play in psychological science. What is conceptual replication? It’s when instead of replicating the exact same experiment in exactly the same way, we test the experiment’s underlying hypothesis using different methods.

One reason conceptual replication is important is that psychologists aren’t always dealing with objectively defined quantities like 2mg of magnesium; instead we have to operationalise conceptual variables in a concrete manner. For instance, if we want to test the effects of good mood on helping behaviour, what counts as a good mood and what counts as helping behaviour? These are not objectively defined quantities, so we have to decide on something reasonable. For example, in the 1970s Alice Isen found that people were far more likely to help someone who had dropped some papers after they had found a dime in a phone booth (Isen & Levin, 1972). But we have even more confidence in this result now that it’s been conceptually replicated: helping has been found to increase not only after finding money, but after reflecting on happy memories, doing well on a test, or receiving cookies. Numerous different ways of measuring helping behaviour have been used as well. In other words, even if nobody had ever tried to directly replicate the original research, the conceptual replications give us confidence that the underlying hypothesis – that a positive mood increases helping behaviour – is correct.

In the recent debate (see tinyurl.com/cfkl2gk and tinyurl.com/7ffztux) about the failure to replicate John Bargh’s finding that priming the elderly stereotype leads people to later walk more slowly as they leave the experiment, there has been a great deal of misunderstanding because of confusion between direct replication and conceptual replication. While it is true that there have been few direct replications of that finding, the underlying hypothesis that ‘activating a stereotype in the perceiver without the perceiver’s knowledge would increase the perceiver's own behavioural tendencies to act in line with the content of that stereotype’ has been replicated many times. It was even conceptually replicated in the very same series of published studies in which the original ‘slow walking’ study was published, as well as by independent researchers at different universities around the world. Taking these conceptual replications into account, most psychologists are not nearly as troubled by a single failure to replicate the result as it may appear that they should be.

In brief, the commentary focuses on the inherent subjectivity involved in addressing psychological questions -- we can't always study human behavior the same way we study physics. Conceptual replication helps us test a hypothesis in several different ways to make sure that we're really testing what we think we're testing. Direct replication is certainly an important part of the process, but if we rely solely on reproducing the same result over and over again then we run the risk of making the same mistakes over and over again, too. Conceptual replication helps us mitigate that risk, as I cover in this post on the history of cognitive dissonance.