Welcome back to the 4th part of the series looking at understanding risk and diagnosis in Emergency Medicine. In part 3 we looked at how tests with apparently high sensitivities (e.g. 98% sensitivities) actually mean that you miss 1 in 50 patients with disease. Now to be honest, if you just look at that figure in the context of a disease like PE or Acute Myocardial Infarction then it makes for pretty uncomfortable reading. I know of few emergency physicians who are immediately comfortable with that kind of figure, and to be honest even as I write it down it makes me stop and think again (and I know the answer to the question that follows).
So how can it be that we apparently miss 1 in 50 PEs in the ED, yet we are not in court every other week on the back of a legal case, or worse in the coroner’s court as a result of a patient’s death? To explain this we need to start thinking and explaining other factors that eventually affect the performance of a test in the real world that we work in. The best way to do this will be for us to start with an example and for ease of convenience let’s stick with Pulmonary embolus. I’ve chosen PE as it is a topical subject and also the PERC rule to help illustrate the issues around risk. This post is not really about PE, more about how diagnostic test work in practice. It’s just a convenient example….., oh and I hope my maths is right as we go along. If its not I’m sure you will let me know 🙂
How likely is PE in the patients I see in the ED?
You should probably have a feel for this from your own experience. What we are referring to is the prevalence of disease in the population of patients that investigate for PE. Probably best if we take out the ones in resus with no blood pressure as to be honest they are not the sort of patients that concern us from a discharge perspective as you are just not going to send them home. Rather, we are thinking about the patient in the ambulatory end of the department who has some respirophasic (pleuritic) chest pain and some mild shortness of breath, but no major risk factors, a normal chest X-ray and a desire to go home (which you share).
The exact prevalence of disease will depend on a number of things but we can use data from the literature to help us so let’s take the incidence from Kline’s paper on the investigation of PE (6.9%) and express that as a diagram based on 100 patients who might turf up in your ED.
The white squares represent patients who do not have PE, and the red ones are those that do. Now of course at this point you have no idea of who has and who does not have the disease so you are going to start investigating them as you would not want to send 6.9% of your patients home with an undiagnosed PE!
Right then!! Let’s get testing.
So, there are a number of things we can do with our group of patients, but it’s pretty clear we need to do something. Since we don’t want to do any invasive tests at the moment we could start with the PERC rule as a way of starting the diagnostic process. (you might also want to whizz over to Life in the Fast Lane for a reminder as well) So what happens if we apply PERC (lets take the original data at a sensitivity of 97.4%, a specificity of 21.9% and a prevalence of 6.9%) to the 100 patients who turn up in the ED? How many of them end up PERC positive and how many PERC negative? Again we can express this as in diagram form as shown below.
Patients represented by the yellow circles (roughly 79% % of patients) are those that are PERC positive and will need investigating further. Those in the green circles (roughly 21% of patients) are classified as PERC negative and are the ones that you might consider sending home with no further testing. [/DDET]
[DDET “But what about the patients who DO have a PE?”] Key question that. We really need to get an idea of where those original 6.9% of patients have ended up after the application of the PERC rule, and again we can get an idea about this by mapping them onto the post-PERC diagram as shown below.
So after applying PERC the patients who do have a PE remain predominantly in the group who need further investigation and that’s fine. However, there is a little bit of red left in the green zone, i.e. in the group that we are considering sending home with no further investigation.
So how many am I actually missing?
If you work the maths out (if you must you can), then it works out that 6.7% of your original 6.9% of patients are in the yellow zone and will hopefully be picked up by further investigation. In the green zone that leaves just 0.2% of patients who had a PE. Why? Well basically your high 90s sensitivity only applies to patients who actually HAVE disease. The sensitivity part of the performance of a diagnostic test is only really relevant to patients with the target condition (in this case PE). So basically, we capture the majority of patients with PE in the PERC positive group.
Great 🙂
That’s just 1 patient missed for every 500 investigated as a low risk PE patient. This is clearly more reassuring. At Virchester we investigate about 700 patients for PE every year so we would expect to be missing 1-2 patients per year if we applied a strategy such as the one described above.
We think that’s pretty good and we will argue that it is an acceptable miss rate. It means that there is a group of patients who do not need to go onto more invasive testing at this stage (so long as you are comfortable with that miss rate). Although we are still missing one in 50 PEs the number of PEs missed per person investigated is actually quite small.
So can I use this in my population?
Well, the key question is really to get an idea about the prevalence of disease in your population. Because of the way PERC rules it will probably be similar, but that’s not the case in other diseases where prevalence can vary a great deal. Clearly, as the prevalence rises you end up missing more patients per group of patients investigated.
In our case study around the PERC rule if we were to increase the prevalence to 30% then we would end up missing more patients with PE (about 5 per 500 investigated) and at some point you would need to say that this is not an acceptable strategy.
If you want to play around with how prevalence changes how your diagnostic test will work according to the number of people investigated then download this excel file and play around with the numbers on a hypothetical population of 1000 patients in your ED. Just input the sensitivity, specificity and prevalence and you can see how it will perform.
In Virchester we really quite like this calculator as to be honest the sums make our heads hurt and it’s quite useful to see how the performance of a test changes with a change in prevalence. We prefer to see the effect as a natural frequency rather than percentages and statistics. It makes it feel more real to us as clinicians and you can also use it with patients if you are so inclined. If you want to work things round the other way by filling in the event rates to see what sens and spec (plus a whole bunch of other stuff as well) then I would suggest using this fantastic online calculator.
This is one of the many reasons why it’s so important to know about the characteristics of your own patients when assessing new diagnostic tests.
What if you miss the diagnosis in a lawyer?
A very cynical question if you don’t mind me saying so! However, it is a fair question as we have accepted that we are going to miss a proportion of patients. We are going to have to balance the convenience of our investigation protocol which does not include blood tests and radiology against the knowledge that some people will be missed. Can we justify this or do we have to accept that we are doing the wrong thing some of the time? I put it to you that the question is ‘what is the wrong thing?’ Have a think and then revisit us for part 5.
vb
S
For all posts in this series click here
Risky Business Part 1 You are not a diagnostician.
Risky Business Part 2 What does a test result really mean?
Risky Business Part 3 Is it acceptable to miss some diagnoses?
Risky Business Part 4 How many patients do we miss (and does it matter)?
Risky Business Part 5 Does a correct diagnosis mean that the therapy will work?
Risky Business Part 6 How many steps to disaster?
Risky Business Part 7. Risk Proximity
Before you go please don’t forget to…
- Subscribe to the blog (look top right for the link)
- Subscribe to our PODCAST on iTunes
- Follow us on twitter @stemlyns
- Like us on Facebook
- Find out more about the St.Emlyn’s team
Like a death star with a tractor beam, VTE has caught you unawares and dragged you slowly and steadily back to its bosom. Awesome.
Just to raise an extra few points, as well as prevalence in our population we also need to consider the local demographic when applying a decision rule I think.
One of the interesting things about the recent PERC systematic review was the variable sensitivity seen when applied to different territories (http://www.emergencymedicine.utoronto.ca/Assets/EmergeMed+Digital+Assets/event/Journal+Club/Diagnostic+Accuracy+of+Pulmonary+Embolism+Rule-Out+Criteria.pdf). Most of the data suggesting high sensitivity comes from studies conducted by Kline himself, often in the US. European data suggests a significantly lower sensitivity (80%). A Similar issue to that discussed in @JC_StE recently regarding external validation of the ABCD2 score in Australia and its poor performance. Is this because other cohorts/investigators apply the rule poorly, as they have less personal stake in its validation? Or are the cohorts actually different?
Likewise, I remember the long discussion we had in JC about a decision rule followed by a decision rule (Wells score and d-dimer) in this context. Are we happy with this? How many decision rules can we apply to one patient with immunity?
Food for thought.
Thanks Dan, and don’t worry, Monty Hall and the d-dimmer dilemma will be making an appearance soon 🙂
Want to co-write it with me?
S
Monty Hall, loved by all.
Always happy to chip in with game theory discussions.
Great post! FYI, the images in “Right then!! Let’s get testing.” and “But what about the patients who DO have a PE?” are not displaying correctly for me, though it could be just me.
Thanks, not sure what happened there, but no fixed 🙂
Thanks for letting me know.
S
Pingback: Emergency medicine, a risky business part 7: Risk proximity - St.Emlyn's
Pingback: When is a Door Not a Door? Bias, Heuristics & Metacognition
Great explanation thanks. Is there a name for the specific statistic that’s calculated at the end? It’s not the negative predictive value of the test.