Does reviewing a paper without seeing its results actually help referees make a fairer decision? The open access journal BMC Psychology, published by BioMed Central, certainly thinks so and will be carrying out a pilot study to find out. This will be the first randomized controlled trial of the new method called “results-free peer review,” in which referees are not shown the result, discussion, and conclusion sections of the manuscripts that they evaluate until the end of the process. The idea here is to reduce publication bias, making reviewers concentrate on the methodology and subject of the study rather than its outcome.
Disregarding Data?
In academic publishing, studies that do not support a hypothesis are usually less likely to be published than those that do; however, these “null” results form an important part of the scientific record. Making such data available to the scientific community could help save time and money because researchers would not repeat unsuccessful experiments and funding institutions would not unknowingly support studies that are unsuccessful. Also, clinical trials could be planned better and patients would not be treated with ineffective therapies.
The Trial
According to Liz Bal, Associate Publisher, BioMed Central, results-free peer review could help reduce publication bias by “basing the decision to publish purely on the scientific rigor of the study design.” The BMC Psychology trial will start with a pilot phase to see if the new reviewing method is viable. After that, a controlled study will begin in which manuscripts are randomly assigned to the “results-free” or normal peer-review process. Initially, the results, discussions, and conclusions of the results-free papers will be hidden from the peer reviewers and the manuscripts will be “accepted in principle” based solely on their premise and methodology. During the second stage, these manuscripts will be reviewed again by the same referees but with the omitted sections visible. The initial decision can only be revoked if the results and conclusions deviate considerably from the original goals.
A Good Idea with Limitations?
A similar trial was carried out by Comparative Political Studies in 2015, showing that although results-free peer review can encourage referees to focus on theory and research design, it raises problems about how to interpret null findings. The 2015 pilot study also suggested limitations to the widespread use of this process in qualitative or interpretivist research. In fact, there are general concerns that the method might not be suitable for evaluating manuscripts outside the social sciences. In physics and chemistry, for example, the results and their interpretation are usually essential to a paper while the experimental techniques and methods can be relatively standard. In such cases, results-free peer review might not work.
The intent to make the peer-review process more transparent has led to many interesting ideas. Results-free peer review is one of them, and although it may have limitations in some areas, it will be interesting to see the outcome of such a trial.
