International Clinical Trials Day 2015 is on 20th May. To mark the occasion we’re going to run an extra special critical appraisal. This time, let’s look at an awesome piece of work I came across in the literature. It’s not exactly hot off the press – but it’s an inspirational piece of work that I’m sure will influence medicine for many years. The topic is also a little off piste but bear with me – I think this one’s a game changer, one way or another…
[DDET What question did they answer?]
In this work, the lone author, Dr. Lind – who was from Scotland – noticed that he had a problem with sailors on his Royal Navy ship (HMS Salisbury) coming down with scurvy and many of them died. He wasn’t sure how best to treat them so he ran a trial.
So, let’s critically appraise this for a minute…
‘Was there a sound rationale for the trial?’
Well, yes, because they had a big problem with scurvy. But I’m not so sure that Lind did everything he ought to in order to justify the position of equipoise. Where’s the systematic review? Where was the early phase work for pharmacodynamics and pharmacokinetics of the interventions? What’s the justification for choosing the interventions he went for? I have some concerns about this.
‘Were the objectives clearly stated?’
It seems to be obvious what Lind was trying to achieve but – no – his objectives were not stated at all, either a priori or in the manuscript. And there’s simply no evidence of a null hypothesis. It looks like Lind wanted to compare six treatments for the treatment of scurvy, which is pretty ambitious.
Was this a superiority trial, a non-inferiority trial, an equivalence trial? He doesn’t tell us. However, I think we’ll let him off as it seems to be obvious that this is, in reality, a superiority trial. The null hypothesis was that there was no difference between the treatments.
[DDET What did they do?]
This was a prospective controlled trial at a single centre (well, a single ship) coming from the UK. They (well, Lind) included 12 patients with symptoms of scurvy. The inclusion criteria aren’t very clearly stated but Lind tells us that “in general” they all had “putrid gums, the spots and lassitude, with weakness of their knees”. No tests were done to confirm the diagnosis of scurvy.
I’m getting concerned here. The inclusion criteria are pretty vague. None of the diagnoses were confirmed and there could have been a huge variation in the severity of disease among participants. In fact, there are no baseline characteristics reported at all – so we can’t tell.
The patients were assigned to one of six treatment groups, as in the pic below. They took their treatment for six days and then outcomes were assessed.
How were patients randomised?
Hmmm. There was no randomisation. They were just assigned to different groups – by Lind. We’ve got another problem here. There’s no allocation concealment so there’s a possibility that the milder cases were selected to go into Lind’s chosen treatment groups. The patients with more severe disease might have been given less promising treatments. Without randomisation, we’ll never know, but we do know that the patients who got sea water (poor things) were “two of the worst patients, with the tendons in the ham rigid (a symptom that none of the rest had)”. Clearly this introduces a huge bias.
[DDET So what was the primary outcome and what were the key findings?]
Well, the primary outcome isn’t actually stated. Not at all. We assume that Lind was looking for recovery but he didn’t define this. We do know that the two patients assigned to eat oranges and lemons recovered quite well – one of them fully recovered, the other was well enough to resume duties after 6 days (whatever that means!). It seems that the outcomes of the patients in the cider group were fairly good. Lind reports, “Next to the oranges, I thought the cyder had the best effects”. Objective? Hmmm. The outcomes of the other groups don’t seem to have been quite as good although the putridness of the mouths of patients in the sulphuric acid group seems to have been reduced.
Without a primary outcome that’s clearly defined (including when and how to measure it), how do we know what success looks like in a trial? Without comparing the groups statistically, how do we know that the findings aren’t just down to chance?
That brings us on to the crux of the whole thing… Lind concludes that oranges and lemon were superior to the other groups for treating scurvy. But let’s take a look at the analysis. There are six groups so I assume that Lind really wanted to do pairwise comparisons – i.e. to compare one group to another and run multiple comparisons. When you do lots of comparisons, you need to be aware that sometimes the p value might be <0.05 just by chance. Sometimes you might choose to adjust for that – and only accept a lower p value. Even without that, what was the p value here?
Assuming that none of the patients in the other groups did well, this is our 2×2 table…
|Patient did well||Patient didn’t do well|
|Patient took oranges and lemons||2||0|
|Patient took sulphuric acid||0||2|
It looks good, right? The absolute risk reduction is 100% – number needed to treat is just ONE! Comparing those two proportions tells us that the absolute difference was 100% BUT the 95% confidence intervals extend from -19% to 100% with a p value of 0.32. So this trial didn’t show any statistically significant difference at all! Clearly a 100% difference is pretty clinically significant – so that means the trial was underpowered. Tut tut.
[DDET What about the way they conducted this trial?]
I think the MHRA and FDA would have something to say about this. There’s no mention of any ethical approval. It doesn’t look like they took written consent from the patients. They might have consented verbally – we don’t know – but did they know what they were letting themselves in for? Were they given a patient information sheet and the opportunity to ask questions? We don’t know. The trial certainly wasn’t registered a priori. And there’s no mention of any declarations, financial relationships with fruitmongers or ownership or orchards. We don’t know which organisation sponsored this trial – or even if the Royal Navy knew about it and gave R&D permission.
[DDET So what are the take home messages?]
All in all, it’s a pretty lousy example of a trial. I don’t think we can take anything meaningful away from this. Should oranges and lemons be used to treat scurvy? Who knows? If I get scurvy, I’ll probably give the oranges a try as I quite like them anyway – and there aren’t too many side effects. I’d quite like to try the cider too, mind you.
Obviously, there is a serious take home message from all this too. This trial is widely held to be the ‘first clinical trial’ by James Lind, reported in 1753. It’s not bad for it’s time – but clinical trials have come a long way since – and just think how much better off we are for it. Happy International Clinical Trials Day! I wonder what clinical trials will be like in another 250 years?
Be sure to check the full text out by hitting this link – it’s an interesting read!