Data Fabrication & Reproducibility: How Triangulation Offers Novel Solutions
Reproducibility has been a standard of good scientific research for many decades. The reason for striving to design a study that can be repeated and produce the same results is quite simple. Other researchers must be able to recreate the results you have observed if these results are to be considered science.
The Reproducibility Crisis
Despite the recognized importance of reproducibility, it is estimated that only about 40% of recently published science can be reproduced accurately. Reproducing scientific experiments has become more problematic for several reasons:
- Pressure to publish: In the rush to gain credit for new findings, scientists are under the pressure to carry out new research, rather than test the reproducibility of studies already published.
- Lack of funding: Because new research is so highly valued in today’s scientific culture, the funds are directed towards this field. There is a corresponding lack of funding for testing work that is already published.
- Complexity of the work: Today’s complex research in biochemistry, physics, and other fields involves huge data sets that are difficult to generate and manage.
- Lack of standards for replication: There are a number of competing definitions for what constitutes a replication study.
As reported in a 2016 survey, scientists who have failed to reproduce the published outcomes of other researchers’ work could have improved the reproducibility by certain measures. The top three ways to increase reproducibility in science would be: better understanding of statistics, better mentoring and supervision, and more robust experimental design.
Triangulation – a New Way to Establish Good Science
Some scientists are questioning whether reproducibility is, in fact, a good indication that the outcome of a project is reliable and has explanatory power. If the hypothesis or experimental design was flawed or biased to begin with, repeating this work does not give us useful outcomes.
Scientists are all taught that “correlation is not causation” but this warning can be forgotten as their careers progress and the pressure to produce practical outcomes builds. Hence when reproducibility is under question, researchers, especially biomedical researchers, feel troubled.
Despite the crisis in reproducibility, the same survey found that most scientists still trust the rigor of published studies. Reproducibility is not the only mark of credible research. An approach called triangulation could serve the same purpose of testing the quality of the research.
What Does Triangulation Look Like?
Triangulation originally referred to the technique of using the location of two known objects to establish the position of a third object, such as a ship at sea. The term has since been adopted in the social sciences to describe a mixture of quantitative and qualitative methods. Triangulation can also be used in the physical sciences. A good description of triangulation is given by Marcus R. Munafò and George Davey Smith in their recent article published in Nature:
“We believe that an essential protection against flawed ideas is triangulation. This is the strategic use of multiple approaches to address one question. Each approach has its own unrelated assumptions, strengths and weaknesses. Results that agree across different methodologies are less likely to be artefacts.”
An example of a triangulated approach in biomedical research is a study by Surana and Kasper. The authors used triangulation to identify therapeutically useful bacteria in the human intestine.
Because triangulated studies often require several different areas of expertise, they could appear quite different in publication than typical papers in a number of ways:
- More multidisciplinary research, involving several areas of expertise;
- Recognition of a greater number of contributors for their work; and
- A larger number of reviewers who will read and assess the study and recommend publication
A further innovation proposed a year ago by Jeffrey Mogil and Malcolm Mcleod, is the concept of the “confirmatory study”. This is a form of rigorous triangulation. In this case, the results of a scientific study are subjected to third party, pre-clinical trials prior to acceptance for publication. The research of new drug therapies could use this approach. The proponents argue that this testing would prevent costly failures in clinical trials. Only the best-designed studies would pass the confirmatory study and become published research.
Will Triangulation Solve the Reproducibility Crisis?
Triangulation is one aspect of the overall global trend towards open research and greater collaboration among scientists. Other aspects of this trend include pre-print servers to share early results before publication and pre-registration of experimental work in advance of the actual results.
Many studies might not survive this kind of scrutiny – and that’s the whole point. The goal is to have fewer published papers of better quality. The research that does make it through the triangulation process will be better designed, produce more reliable results and possibly lead us faster to effective applications. Using different methodologies or having an outside party assess the research is helpful. This is because it forces researchers to return to study design principles and more emphasize more on hypothesis testing.
These new approaches to open science—like triangulation and pre-print servers—do not replace reproducibility. With openness in research comes responsibility and mutual testing, good practices that have always strengthened science.
How Can You Stay on Top of this Trend?
Triangulation is still a new concept in the scientific community. Its definition and its implementation will continue to change. Moreover, there are a number of things that you can do to follow the discussion and contribute your own opinion in the debate.
- Compare your views on reproducibility with the views of other scientists.
- Refresh your skills in experimental design and hypothesis testing by reading research papers outside your discipline.
- Contribute to online discussions about open research, including topics like triangulation and confirmatory studies.
Have you tried the method of triangulation for any study of research? If not, how far successful do you think the method will be? Please share your thoughts in the comments section below.
