High-quality scientific manuscripts should include accurate and rigorous statistical and data analyses, allowing other researchers to reproduce the findings presented. Although most academics strive to collect, analyze, and publish their results in a clear and reliable manner, mistakes may occur at any stage of a study.
This can be detrimental not only to the researchers themselves—or to the journals in which the findings have been published—but also to the scientific community as a whole. This is because it contributes to the so-called “reproducibility crisis”, which involves great amounts of published information that cannot be confirmed by others. Thus, checking the statistical analysis in scientific publications is a crucial part of the peer review process. In addition, this acts as a key step to ensuring and enhancing the quality of academic papers. This process of statistical analysis is called statistical review.
Peer Review: A Great Responsibility
Peer review is both a privilege and a great responsibility. A good peer reviewer should assess the importance and originality of a study, correctly evaluate how the collection and analysis of data, and present all the information in a clear and understandable way. This means that referees must recognize the limitations of a particular method or strategy and check the rigorousness of the statistical analyses as well as the use of appropriate controls.
Unfortunately, young investigators, however, do not receive any formal training in peer review. Hence, they have to learn through experience and mentoring. In addition, in most cases, referees do not have the experience required to evaluate the statistical methods used in a study so it may be necessary to consult a statistical expert. If you would like to improve your peer-reviewing skills, here’s a short guide to becoming a good peer reviewer.
Importance of a Good Statistical Review
Incorrect statistical data slows down scientific progress, wastes research funding, and can result in retractions. A faulty analysis may include errors in measurements, failures during study design, oversight, or even outright fraud. In some cases, it can also be the result of unrealistic expectations on the part of investigators.
To solve this problem, the journal Headache recently implemented a reviewing model that includes a careful statistical evaluation. Since 2009, statistical and methods reporting are checked for reproducibility and validation. The decision letters now include a detailed assessment of the statistical analysis and an evaluation of the design and methods involved in the study. Also, authors must upload relevant reporting guideline checklists. The Annals of Internal Medicine also offers an exhaustive statistical review process, while BMC Medicine relies on subject reviewers to indicate when a statistical review is required.
What to Look for During Statistical Review
Statistical assessment has become an important part of the editorial process. Some journals send all manuscripts for statistical review whereas others only do so if the methods are particularly complex or the editor(s) and/or referees have concerns. However, what should evaluators look for during a statistical review?
- Is the hypothesis clearly stated? Are there any deficiencies in previous research highlighted in the Introduction?
- Have relevant reporting guidelines (CONSORT, STROBE, etc.) been followed?
- Have the methods (sampling design, inclusion/exclusion criteria, etc.) been described in detail? Would it be possible to reproduce the results of the study?
- Are the methods appropriate for the study question and study design?
- Have the authors presented enough information to verify their calculations and conclusions?
- Does the article mention any missing variables or missing data? Have all the limitations of observational studies been considered?
- Are any results interpreted beyond the range of the data?
- Does the study comply with ethical guidelines? Are there any conflicts of interest to consider?
- Are the Conclusions supported by the data presented in the Results section?
- Since the reproducibility and validity of the results are crucial, the authors should also be willing to provide the study protocol, statistical codes, and appropriate references for the methods used if required.
How important do you think statistical review is? Besides the ones mentioned above, do you think there are any other key points that must be kept in mind during a statistical review? Please share your thoughts in the comments section below.
