What Should You Do When You Can’t Replicate a Study?

A Process Based on Assumptions!

When a replication study is commissioned, much of the initial work is based on a series of assumptions:

• The initial study was conducted by experienced researchers with appropriate oversight
• The research paper was given a thorough peer review by experienced researchers prior to acceptance for publication (although you are unlikely to know what, if any, revisions were requested prior to publication).
• You were given access to all relevant data and materials to replicate the study.

A Battle of Reputations

When errors are found, a deliberate sequence of events will be initiated that will place the academic track records and reputations of all stakeholders under scrutiny. Even if the team leading the replication study has a reputation that is perceived to be higher, based on prior work and experience, the onus will be on that team to double and triple-check their work prior to even raising the issue of concerns over the accuracy of the original study. Since the original paper has been peer-reviewed and published, the authors hold the higher ground of validity until proven otherwise, and since retractions carry such a negative connotation; the journal editor is likely to back that paper until conclusive proof is delivered to the contrary.

The Problem of Motive

Discovering an error, or series of errors, is only the beginning of a long and often drawn-out process of further investigation. What kind of error is it? Is it attributable to simple human error or sloppy research, or is there evidence to suggest that there was a more deliberate attempt to fabricate results? None of those questions will receive definitive answers until you are sure that you have the full data and materials from the original study. You may have been led to believe that you had all that when you started your study, but it’s likely that misplaced or forgotten details may well materialize as a justification for the questionable results. Whether that was a deliberate withholding of proprietary material or a simple error is, again, up for further investigation. That, of course, assumes that the original authors will even respond to your enquiries and that the journal gives you permission to contact them directly.

An Anticipated Outcome

By definition, good research must be accurate and replicable. The Scientific Method assumes that errors will be caught through a constant process of testing and refinement. The current lack of interest in replication studies seems to have forgotten that. Replication studies have a hard time getting published, which directly impacts the availability of funding for such studies, and the frustrating and unproductive process that a failed replication study must then endure to investigate what or whom is at fault can be traumatic enough to convince any researcher to pass on the opportunity.

If the replication study has been thoroughly vetted and the erroneous results still stand, shouldn’t the journal and the original authors do the right thing and work with that study team to resolve those errors? Unfortunately, the reputations of stakeholders often seem to be of greater concern than the truth of the matter.