Reproducibility is vital to research practice. Fundamentally, it means that different researchers use the same method, under the same conditions, and they should get similar results. Reproducibility is the only thing that an investigator can guarantee about a study. In addition, it ensures transparency and helps in understanding the research methods used in the study.
Research published in journals should be reproducible. However, many articles do not provide enough information for the work to be reproduced elsewhere. Leaving out important information also means that reviewers cannot assess whether or not the study is reproducible. This dramatically reduces the scientific value of the work.
What is Preproducibility?
Until recently, there has been no word to describe research that is not reproducible due to a lack of information. In an article in Nature, Philip Stark suggests a new term: preproducibility.
An experiment is preproducible if the author has given enough accurate information for others to repeat it. Stark compares a preproducible report with a recipe. He explains: To bake a loaf of bread, we need more than just a list of ingredients. We also need to know how those ingredients are combined and treated. Preproducibility can be described as a “scientific recipe.” If the recipe gives enough detail, anyone should be able to follow it and get a similar result.
What Happens When Research is Not Preproducible?
In some areas of science, as little as 25% of research papers are reproducible. This can have serious consequences. Failing to include enough detail in a report can waste researchers’ money and effort. In medicine, irreproducible research can delay treatments, as they cannot be proven. This wastes the valuable time of both patients and doctors.
Peer reviewers struggle to review papers that are not preproducible. How can reviewers assess an article when it is impossible to check whether the work can be reproduced? As Stark says, “Science should be ‘show me,’ not ‘trust me.’” At worst, research that is not preproducible could destroy confidence in the scientific process.
How to Report Research
Research must be reported in a systematic, organized way to ensure preproducibility. This helps reviewers to evaluate the work and allows other researchers to reproduce the study.
Stark says that he will always publish his raw data (where possible) and the software he has used. This is an excellent start.
Stark also suggests several points that researchers should include when writing a paper/thesis. These should answer any questions a reader might have about how to repeat the research.
Some of the points to consider:
- Have you included your raw data? How did you collect it?
- Are your materials and method detailed enough to allow repeat data collection?
- How did you process your raw data?
- Is your data analysis reproducible?
- What ad hoc choices did you make, and why?
- Why did you choose your method of analysis? Did you consider others?
- Have you included any code and data needed to reproduce tables and figures?
These are just a few of the questions that researchers should consider when writing a preproducible report. Exactly what needs to be included will vary, depending on the research field and the type of study.
Challenges for Reproducible Research
Researchers often experience pressures that can hinder good scientific practice. Most individual scientists want to produce work with long-term value. However, they often face pressure to produce high-profile, “flashy” results instead. According to an article in Nature, “the incentives to be first can be stronger than the incentives to be right.”
While higher salaries might tempt researchers in the industry, the pressures on academics can be varied. Publications, promotions, awards, and better funding prospects can all lead to “bigger,” rather than better, research. Most seriously, all these pressures can tempt scientists to ignore or re-work “undesirable” results.
So, what is the solution? Some organizations are taking steps to tackle the problem. Journals, including Nature and Science, have updated their guidelines and introduced checklists. They inquire whether researchers have used good scientific practices such as randomizing, blinding, and calculating the right sample size. The US National Institutes of Health invites reviewers to decide whether experiment plans are soundly based on the details provided by the researcher
Unfortunately, the research institutes have not yet come streamlined the research sharing process. To support strong science, universities could follow Good Institutional Practices (GIP).
Good Institutional Practices(GIP)
For some practices, such as animal testing and clinical work, researchers must be licensed and receive regular training. Institutions must have a member of staff to check that guidelines are met, who in turn is accountable to a regulator. This process ensures that the work meets very high standards.
GIP could be the basis of a similar system for all research. This would help to support good scientific practices and avoid research that cannot be reproduced. If funding was linked to meeting GIP, institutions would be encouraged to comply.
How to Meet GIP
In summary, GIP should include these points:
- Discussion of research methods; Institutions should have formal processes in place to discuss methods and data collection. Researchers within the same institution should assess each other’s work.
- Reporting systems; should allow everyone, from students to staff members, to formally report any concerns with research practices.
- Training and standards; Often, researchers are assessed on the number and impact factor of their publications. Instead, they should be judged on their knowledge and application of sound scientific practices.
- Records and quality management; All records should be available for independent review.
- Incentive and evaluation systems; These should aim to encourage those who follow good scientific practices and excel as mentors.
- Failures to meet GIP should be recorded and investigated.
Researchers, funders, and journals can help to encourage GIP. However, achieving GIP must ultimately come from institutions and reproducibility, and preproducibility should be a common practice.
What do you think of the new term “preproducibility”? Is this something you already aim for in your work? Let us know your thoughts and experiences in the comments below.
