Simonsohn's Fraud Detection Technique Revealed

urisimonsohn

Uri Simonsohn's "secret" paper describing the analyses he used to detect fraud in the Dirk Smeesters and Larry Sanna cases has now been submitted for publication and is available on SSRN. It's titled "Just Post It: The Lesson from Two Cases of Fabricated Data Detected by Statistics Alone." Simonsohn explains the analyses he used to detect and confirm the fraud and calls on journals to make the publication of raw data their default policy. Here's the abstract:

I argue that journals should require authors to post the raw data supporting their published results. I illustrate some of the benefits of doing so by describing two cases of fraud I identified exclusively through statistical analysis of reported means and standard deviations. Analyses of the raw data provided important confirmation of the initial suspicions, ruling out benign explanations (e.g., reporting errors; unusual distributions), identifying additional signs of fabrication, and also ruling out one of the suspected fraudster’s explanations for his anomalous results. If we want to reduce fraud, we need to require authors to post their raw data.

Simonsohn also explains the measures that he took to avoid making false accusations, including replicating his analyses across multiple papers by a single author, analyzing the raw data, and contacting the authors directly about his concerns, and only then expressing his concerns, discretely, to those who are entrusted with these sorts of investigations. The paper makes it very clear that Simonsohn took a very thoughtful and conservative approach at every turn and took great pains to rule out the possibility that there could be a perfectly innocent explanation for the statistical anomalies he discovered. Here are the measures he took in point form:

1. Replicate analyses across multiple papers before suspecting foul play by a given author,
2. Compare suspected studies to similar ones by other authors,
3. Extend analyses to raw data,
4. Contact authors privately, transparently, and give them ample time to consider your concerns,
5. Offer to discuss matters with a trusted statistically savvy advisor,
6. Give the authors more time.
7. If after all this suspicions remains, convey them only to entities tasked with investigating such matters, and do so as discretely as possible.

I'm sure there will be a lot of discussions of the techniques Simonsohn employed and whether more fraud will be uncovered by Simonsohn or others in the weeks and months to come. As I argued here,fraud is a problem that psychologists need to take very seriously, but no less seriously than the much less egregious misdemeanors that are committed, often unintentionally, on a regular basis. Luckily, we have someone like Uri Simonsohn leading the charge on both fronts. Hopefully journals will take action on Simonsohn's recommendation to publish raw data, as well as some of the solutions he put forward with his co-authors in Psychological Science last year aimed at reducing false positive findings. For more discussion other important reforms to consider, see this list compiled by Chris Chambers.

 [notice]Update: see this post for additional information[/notice]