As an academic, you rely on publications to secure and further your career. However, publishing a research is not easy. For decades, academics have questioned the way journals select and review the articles that are published. Some journals have responded with publishing the peer review reports they receive from referees. This has raised another concern: will an open peer review system affect the way referees write a review? Researchers performed a pilot study to assess this. Let us find out more about this study.
Pilot Study of an Open Peer Review System
A pilot study was conducted involving five top journals to evaluate whether publishing peer review reports would affect a referee’s behavior. Following factors were considered:
- Reviewer’s willingness to review articles.
- The type of recommendations the reviewers made.
- Reviewer review turn-around times.
- The tone of the report.
Following these criteria minimized the bias:
- Journals selected had similar impact factors, number of submissions and submission dynamics.
- Comparison of reviewer behavior for reviews conducted before and during the study. In addition, the control group for the same used the closed peer review system.
- Well-informed authors and reviewers.
- Reviewers agreed to publish their reviews.
- Reviewers had the option of remaining anonymous.
- The results included only first round reviews to minimize the effect of reviewer learning.
- The study included both open-access and subscription-based journals.
Pilot Study Findings
Statistical analysis of comparison of reviewer reports before and during the pilot study gave the following outcomes:
- Willingness to Review: Knowing that their review would be published did not put referees off reviewing papers. Although the referee’s acceptance rates to review a paper decreased from 43.6% to 30.9%, even the control group showed similar results. Therefore, there was a general decline of review acceptance, not influenced by the trial.
- Recommendations: Overall, making review reports public did not bias recommendations made by reviewers. The only differences noted were within referees of different status. Younger and non-academic referees tend to be more positive than professors and PhDs.
- Review time: Previously, reviewers did not pay much attention while writing their reports. With their review reports published laters, referees may spend more time on the language and structure of the report.
- Tone of the report: As the review reports will be reaching a wider audience – not just the editorial board of the journal – the tone of their report was expected to be different. The tone of the report observed was slightly more positive tone, but the differences were minimal.
Open Peer Review System Considerations
Publishing peer review reports seems a workable solution to improve the transparency of the peer review process. A few points to think about are:
- This was a pilot study and further research across all disciplines needs to be performed.
- This method of transparency could be more effective than pre-publication interaction between referees, editors and authors.
- Anonymity emerged as an important factor. Reviewers were far more comfortable with their reports being published if their identity was not revealed.
- Identifying reviewers protects journals from accusations of conflicts of interest.
Benefits of Publishing Peer Review Reports
Publishing peer review reports could be a great tool for everyone in the research community. In addition to the benefits mentioned above, it could help researchers improve the quality of their papers by learning from review reports. Consequently, journals could benefit from receiving submissions of a higher quality. The entire research community should welcome the move to publish peer review reports. Do you agree? Let us know in the comments section below.
