Imagine you are a young, keen PhD student. Deep in the research for your thesis, you decide to take a break by browsing the latest news in your field. That’s when you spot a story on article topic similar to your own research: the paper has been retracted by the journal due to scientific misconduct. You begin to wonder – could this be avoided? Is there any way for the scientific community to check the accuracy of published data? Can researchers avoid retraction by double-checking their own data? In fact, the scientific community does have two methods to identify errors in research.
GRIM and SPRITE
This is when you might discover GRIM and SPRITE. Both methods were developed by researchers James Heathers and Nick Brown. The two scientists had an idea for something many others wished they had thought of: a simple test to evaluate the accuracy of published research.
Heathers and Brown first developed Granularity-Related Inconsistency of Means – better known as GRIM. Because GRIM and SPRITE do not need the complete data set, they can be carried out by anyone with access to the paper.
How Does GRIM Work?
GRIM tests whether a reported mean is possible, given the sample size. It works on studies where data is collected in the form of whole numbers. For example, participants’ ages in years, or opinions on a scale of “1 = Strongly agree, 5 = Strongly disagree.” With this type of data, the mean has strong granularity (hence the “G” in GRIM.) This tells us that the mean is made up of individual parts that are not continuous.
Heathers gives a useful example. If twelve people are asked their ages (in whole numbers), the smallest amount that the mean can change by is one-twelfth. We can work this out without knowing what the actual ages were. A reported mean of 20.95, for example, is impossible if the sample size is 12.
GRIM was developed by looking at data that the researchers knew was incorrect. Heathers and Brown looked at 260 articles from leading psychology journals. Of the 71 that were suitable for testing with GRIM, half reported mean values that were not consistent with the sample size. Over 20% contained more than one inconsistency.
A simple, free online GRIM test is now available. GRIM is suitable for studies where the data consists of small samples of whole numbers, so it is most useful in the social sciences.
What Does SPRITE Do?
GRIM can clearly be very useful, within limits. In 2017, Heathers introduced a new test called SPRITE (Sample Parameter Reconstruction via Iterative Techniques.) Like GRIM, SPRITE is a simple test to detect inaccuracies in published data. SPRITE does this by recreating potential datasets using only basic summary statistics. This includes the mean, the standard deviation, the sample size and the range of values.
The advantage of SPRITE over GRIM is that it can be used with larger sample sizes, and can therefore be applied to more studies. SPRITE can also spot studies where the summary statistics are possible, but suggest a highly skewed or unusual dataset.
SPRITE creates a large number of datasets that could potentially fit a set of summary statistics. First, SPRITE generates a dataset with the correct mean. It then adjusts random pairs of values in the sample until the target standard deviation is reached (the “iterative” part of the name.) Finally, common sense can be used to work out whether the results are realistic.
Heathers gives an entertaining example. In a study looking at how many carrots school children eat, the summary statistics were tested with SPRITE. The results said that the children were eating, on average, 60 carrots each for lunch! Needless to say, the results obtained showed incorrect data.
Identifying Scientific Misconduct
As the creators are keen to point out, GRIM and SPRITE detect inconsistencies. They cannot tell us why a dataset is wrong, sniff out scientific misconduct or lead to retraction of a paper. However, used carefully, there is no doubt that these techniques are potentially valuable tools in seeking out problems in research. That young, keen PhD student can be reassured.
