Why Don’t Journals Like Replication Studies?

Validation is Not News

Academic journals face something of a quandary in their publication cycles. Publishing monthly or quarterly gives you the luxury of curating your content more carefully than scrambling to find verifiable sources in a newspaper working on a hard print deadline. However, that luxury is quickly lost when you consider the extended time from submission to publication of a research paper or article. If the researcher was attracted to a “hot” topic, time has already passed in the conduct of the research and the subsequent writing of the paper. Add that to the preparation time for publication (assuming the paper is accepted with minimal revisions from the first journal to which it was submitted), and the likelihood that the topic is still “hot” is greatly reduced.

Ironically, academic journals have responded to the increased speed of digital online publishing not by increasing their own internal processes, but rather by steering towards more counterintuitive content. Former bastions of leading research now seem to be willing to embrace elements of Ripley’s Believe It Or Not, in order to garner attention and maintain citation volumes at the appropriate level to keep their JIF rankings. For this reason, validation studies that are completed months after a research paper has been published stand little chance of finding a warm welcome in the same journal. Validation is not news.

Staying Out of the Fray

One critical element in this lack of interest in replication studies is the inherent assumption that all research can be trusted. There are enough mechanisms in place, it is believed, to ensure the accuracy and integrity of the research and the researchers for everyone to assume that the research is valid until proven otherwise. If a replication study fails with a consequent inference that the original study wasn’t valid, journals have developed a response procedure that casts doubt on the replication study authors first, before any questions are asked of the original authors. If those original authors elect not to comply, journal editors will still tend to err on the side of the original study and make a decision as to whether the replication study is valid enough to pursue the issue further.

Money or Validity?

The primary motivator here, of course, is financial. As more money flows into applied research, disputes over the validity of research studies can quickly degenerate into expensive litigation. For the journal involved in publishing work related to that research, there can be several landmines to avoid—alleged failures in the pre-publication peer review, alleged failure to respond to notifications of concerns based on a failed replication study, and alleged failure to pursue authors who chose not to respond to notifications of the results of that replication study, or not taking action to retract the article based on that failure to respond.

Embracing a More Expansive Obligation

Academic journals profess their roles as proponents of scientific knowledge (often in response to criticism over how long it takes to get an article published). If accuracy and integrity are too important to rush the publication process, shouldn’t there also be an obligation to proactively support the validation of the research that is published? The wafer-thin market for replication studies now impacts the interest in funding such studies. If there’s less money to go around, why waste it on research that may not see the light of day in an academic journal?

Offering space to one or two replication studies in each edition or even taking a more community-oriented approach and creating a separate online venue for replication studies without a pay wall, would go a long way to re-affirm that journal’s commitment to research validity and integrity, and it might even convince the eager researchers awaiting an answer on their submission to be a little more patient.