Keep it Transparent: Reliability in Clinical Data

Ninety trials published over the last 15 years include incorrect data, according to a recent review.

Analysing 5,087 clinical trials, the review found a significant number of statistically unlikely data patterns across 8 major journals, suggesting that the information has either been recorded erroneously or purposefully falsified.

This raises significant questions about reliability in published papers, leaving pharmaceutical stakeholders and patients understandably shaken.

Published in 8 leading US medical journals, these incorrect statistics are proof that even the most stringent of publications can allow errors to slip through the net, lessening public and professional trust in established scientific sources.

We explore what this review means for the scientific community at large, as well as how companies can reassure their stakeholders in the wake of this discovery through transparent communication and reliable data.

Error or falsification?

The study, carried out by John Carlisle of Torbay Hospital, used statistical tools to compare baseline data to known distributions of the same variables in a random sample of trial populations.

This method allows researchers to flag up any significant differences between statistical expectations and the baseline data presented in the study. These anomalies could signify data tampering, as fabricated data sets are unlikely to match the same random variations found in a true trial sample.

Of the 90 trials that Carlisle marked out as having skewed statistics, 43 contained data with a one-in-a-quadrillion chance of occurring naturally. The verdict is currently out whether all of these data sets have been actively tampered with, or whether they are down to misinterpretation, statistical error or minor errors such as incorrect transcription of data.

Cambridge’s David Spiegelhalter warned against jumping too quickly to the conclusion that the data had been purposefully falsified, stating, “just because fabricators had a good chance of producing non-random data, does not mean that non-randomness means a good chance of fabrication.”

Nonetheless, the review has prompted serious discussion and investigation around the reliability of clinical data, and how researchers and publishers alike can better screen their studies before presenting them to the public.

The effect on patients

Scientific integrity aside, the most significant concern arising from Carlisle’s review is how falsified data can affect the patients who rely on accurate research to manage life-altering illnesses.

Carlisle highlighted this potential adverse effect, stating, “innocent or not, the rate of error is worrying as we determine how to treat patients based upon this evidence.”

Andrew Klein, editor-in-chief of the Anaesthesia journal in which the review is published, added that all the studies identified by the review should be investigated immediately, as there is a chance that the treatments developed or recommended by such studies are less effective or more risky than scientists have been led to believe.

In such cases, treatments will need to be withdrawn from use as soon as possible in the interests of the patients who consented to such treatments without a full and correct knowledge of the risks and effects.

Avoiding error

Armed with the knowledge that significant amounts of published data has the potential to be incorrect, scientists are considering how we can avoid future error and tampering.

In the short term, Carlisle’s paper has served to ‘name and shame’ the authors of the 90 highlighted trials, with relevant journal editors being informed of the errors ahead of time. Six of these journals have already committed to raising in-depth investigations into cases that could not be adequately explained by the authors in question.

In future cases, journals may take a cue from Carlisle himself, adopting similar statistical screening techniques across all accepted studies before publication. With a wide availability of data and analytical tools, Klein states that there is “no excuse” for any medical journal not to test statistics for accuracy.

Programs such as the controversial Statcheck will likely come into their own in light of Carlisle’s study, capable of weeding out everything from rounding errors to more serious scientific misconduct.

While Statcheck’s aggressive methods have come under fire (within 24 hours, the program automatically informed the authors of 50,000 published papers of their mathematical errors, posting the details publicly), it is a prime example of the power of AI to catch statistical mistakes before they’re published.

Moving forward

In the meantime, it’s important for pharmaceutical companies to respond to Carlisle’s research in a way that reassures patients, practitioners and financial stakeholders.

The study has come at the time of a wider push for more open, accountable and transparent science, championed by developers of data analysis software such as Statcheck’s zealous creator Chris Hartgerink.
The latter is pushing for scientists to make all of their data public and transparent throughout their studies, registering their intentions before they begin experimenting to eliminate post-hoc changes in reasoning, and publicly checking their statistics both before and after publishing.
While such strict measures would drastically undermine the current privacy and freedoms of pharmaceutical research, making it harder than ever for companies to achieve exclusivity in their drug blueprints, rethinking transparency is crucial to preserving public trust in the scientific community.

The open source model offers one transparent solution, allowing researchers from around the world to collaborate openly on projects, with all research freely available online. While this approach has proven successful in cases of socio-economically linked diseases such as malaria, it may be some time before the biopharmaceutical community as a whole is able to embrace this way of working.

In the meantime, it falls to individual firms to underline their commitment to transparency and reliability, taking advantage of technological advances to check their own data as they work, and using the results of such analyses to reassure the public of a solid core of scientific integrity in the pharmaceutical industry.

How do you think we can improve the reliability of data in published clinical trials? Join the debate on our LinkedIn page.