In the highly competitive science race, journal impact factors (based on citations) are often used as indicators of journal rank and therefore, scientific excellence. However, this research metric has been manipulated and abused by authors, journal editors, and publishers who artificially boost citation numbers to help them come out on top. This is done through self-citations, publication of mini reviews and letters, and not citing the first clinical observation or scientific concept. There is no system in place to ensure reproducibility or transparency. This has made the impact factor a hugely unpopular journal rating system.
“It is common, and encouraged by many journals, for research to be judged by the impact factor of the journal that publishes it. But as a journals score is an average, it says little about the quality of any individual piece of research.”
Randy Schekman
Transparency and Openness Promotion (TOP)
Welcome to the TOP factor – a way to counteract the inappropriate use of the journal impact factor (JIF). The TOP factor promotes transparency and reproducibility. This rating system is transparent and non-profit. However, the shift to the TOP factor may not be that straight forward for scientific editors.
Change Can Be Challenging
There will be extra administration in ensuring the new standards are put into practice. Journals will have to put new policies in place to meet the standards as well as provide ways for authors to adopt these policies. Databases for the provision of datasets will be required as well as declarations that authors have complied with the policies. Editors will likely want to conduct audits on studies that they publish to ensure they are compliant. This might mean a longer review time before an article can be published.
Aligning Scientific Ideals with Practices
The TOP factor has eight transparency standards. Journals who wish to adopt the TOP factor, must select one of three levels for each standard. Level one is the least stringent and level three the most stringent. If a journal does not meet a criterion, they score 0. The journal scores higher if they require higher levels of stringency from researchers. The different levels allow journals to adopt the new ranking system in stages. The eight transparency standards are as follows:
Standard 1 – Citation standards: Like the impact factor, this standard requires original intellectual contributions to be given credit.
Standards 2 – 4 – Data, Analytic methods (code), and Research materials transparency: The process is similar for these three standards; therefore, they are grouped together. These standards require the article to state whether the data, code, and materials are available, including information on how to access them.
Standard 5 – Design and Analysis transparency: The research design and analysis process must be fully disclosed. Readers must be able to identify the reporting standards a paper was subjected to and what the paper is disclosing.
Standard 6 – Preregistration of studies: This will make studies that are not ultimately published more discoverable.
Standard 7 – Preregistration of analysis plans: Confirms the difference between confirmatory and exploratory research. The study design, variables, and treatment conditions must be preregistered. The analysis plan will detail the statistical model that will be used in the study.
Standard 8 – Replication: This is a fantastic standard because of the replication crisis. Depending on the level selected, independent replications of the study are either requested or mandatory.
These eight standards are more aligned with the quality of a journal paper, rather than a popular one. Currently, over 250 journals are presented on the TOP factor website and it seems many journals are welcoming it, especially in fields such as psychology, economics, education, and general science outlets.
Impact vs. TOP – What’s the Difference?
You’re probably thinking, all these standards are what researchers and publishers are supposed to be doing anyway. I agree. It is sad to think that there must be measures in place that force scientists to do the right thing. However, the new criteria offers something more:
- TOP does not only focus on novelty or newsworthiness
- TOP is a rating, not a ranking
- The TOP scoring system is transparent and freely available on its website
TOP Is Not Perfect
Some limitations of the TOP Factor include:
- No enforcement of policies
- It is still a journal rating system and does not rate individual articles
- Some studies do not need to be reproducible by nature, such as history. These journals would score unfairly lower than data-driven studies
- Just because a study is transparent doesn’t mean it is of good quality
These limitations can be overcome, as stated by Chris Chambers:
“In academia, culture is the shadow created by the machine of rules, norms, mandates and incentives that drive everyday decisions. If we want to fix the machine, it makes no sense to direct our efforts at the shadow. We must instead replace the parts, one by one, and eventually – if necessary – the entire machine. If we succeed, the culture will have changed, but only because we changed everything else.”
Chris Chambers
A Step in the Right Direction for Research
The Journal Impact Factor is an easy way to rank journals and the quality of the articles they publish. However, as mentioned above, this research metric can be easily skewed. The new TOP factor should solve the issue of transparency and reproducibility. Although transparency does not ensure quality, it gives editors a way to evaluate the quality of the study. It will be an adjustment for publishers to change and implement these policies, and this will take time. TOP still only evaluates the quality of the journal, and not the individual research published in it. In addition, there are no plans to audit journals at this stage. These issues could be addressed down the line. For now, the assumption is that if a journal is rated highly, they are likely to only publish good quality research and we should trust the editors’ and reviewers’ opinions in this regard. The introduction of the TOP factor is, however, a step in the right direction.
