I was recently reviewing a number of documents relating to outcomes of training for one of my other jobs at St.Emlyn’s. There is a lot of interesting data out there about how well trainees perform in professional exams and also during their end of year training assessments (the ARCP).
Anway, most of this data is publicly available but perhaps not widely read as it can be a little challenging at times. However, I was particularly interested to look at data from the recent review of training in Emergency Medicine by the GMC and also from the web published Annual Speciality report from the GMC. Together with the Annual Speciality report from the College of Emergency Medicine they make for some really interesting reading regarding the future of training in the UK.
Why is this important?
It’s important because EM faces a recruitment crisis in the UK and it is possible that the examination/qualification route may be a reason for this. The current model of training requires our trainees to pass many exams.
MCEM – Divided into 3 parts. All must be passed
- Part A – Basic Sciences
- Part B – Short answer questions
- Part C – OSCE
FCEM – Divided into 5 parts. All parts must be passed. Critical appraisal component now taken earlier than other three parts. SAQ and OSCE must be passed at the same sitting.
- Critical Appraisal
- Short answer questions
- OSCE
- Management viva
- Critical Topic review
That’s a LOT of exams to be taken in a relatively short training program and it seems to me that my trainees are always studying for an exam from the moment they choose our speciality to the point at which they eventually pass FCEM. This is not a happy journey and anyone who works with trainees cannot help but feel that there might be a better way.
I’m interested in this after looking at educational outcomes across many specialities as part of my other roles in education. There is some fascinating stuff out there (if you’re into that sort of thing), and at St.Emlyn’s we obviously took a microscope to the educational outcomes in UK emergency medicine. We were surprised, and worried about what we found and thought we should share the data that’s out there, and also look to see how changes to the structure of the exam might improve the educational outcomes for our trainees…….but before we consider the future, let’s just have a think and a delve into the current situation and ask ourselves why we might need to change…..
Disclosures
- I’m an FCEM examiner. I did FCEM and passed first time when the exam was very different.
- I try to help trainees get ready for the exam and have written on the subject.
- I became an examiner when the consultants in our department complained that we did not understand why our trainees were failing the exam. Becoming an examiner has (I think) made me a better trainer and I would recommend all consultants to get involved. It takes time and effort but it’s very rewarding.
- I have received expenses (travel/hotel) as an examiner.
- I have helped write and review aspects of the exam in recent years.
- I believe that the current examiners and college are doing a good job in delivering the current exam system.
- The examiners I have worked with have been excellent. They are highly motivated and put a huge amount of effort into the exam. This is often in their own time and at some personal expense in terms of time and energy.
- The views expressed here are entirely my own thoughts and in no way represent a college/hospital/training program view.
Let’s look at pass rates.
There are two things that I want to know about pass rates. The first is how many people get through the exam and the second is how many get through at their first attempt.
The data for the first of these questions is out there. If we look at data from the Annual Speciality review and from recent GMC education review in EM we can get the following information.
Overall FCEM pass rates.
- 46.7% in 2011-12
- 48.9% in 10-11
- 50.8% in 09-10
- 60.3% in 08-09
This worries me a great deal as this means that at the time of sitting, which is by definition at a point where enough training has been completed to be eligible for the exam then fewer than half the candidates pass, and the results are getting worse. We are looking at a year on year decrease in pass rates and we need to think hard about why as this can only be demoralising for the candidates and in all honesty quite frustrating for trainers.
@CEMpresident @Cemdean
Overall FCEM pass rates.
46.7% in 2011-12
48.9 in 10-11
50.8 in 09-10
60.3 in 08-09http://t.co/CegfidHris— Simon Carley (@EMManchester) August 13, 2013
Reason 1: Is the exam getting harder?
I don’t think so. The format has remained the same over the last four years and if anything it should perhaps have become easier.
Trainees and trainers should be more familiar with the format, will have received feedback from past FCEM’ers and examiners and there are only so many questions that can be recycled back and forth into the test.
Greater familiarity and an ironing out of ‘bad’ questions would arguably lead to an increase in pass rates so I don’t think this is the reason
Reason 2: The trainees are not as bright
Now. This could be a tricky one….are the trainees less clever than they were?
No, certainly not the ones I deal with anyway. The College dean has suggested that there are a greater numbers of doctors taking the exam from non-traditional training programs (at least that’s what I think he is suggesting), and that this may reduce the overall quality of candidates.I cannot comment because I don’t have that information available, but it is possible that the type and preparation of candidates from then and now is different and that this is reflected in the results.
If that is the case then I am quite perturbed. I don’t really like to divide our junior doctors into trainees and non-trainees. All doctors are trainees from the day they decided to do medicine and we do our non-deanery appointed colleagues a dis-service to think otherwise (I’m sure @CEMdean thinks same in practice)
Reason 3: Gamesmanship
The FCEM requires a candidate to pass all 5 parts of the exam, but not at the same time. Alth0ugh the clinical components need to be passed at the same sitting (OSCE and SAQ), the other elements can be passed at different times. That means that if you have the time (and the money) then you can approach the exam in a modular fashion. To some extent this already happens as candidates can take the critical appraisal component from year 4 onwards and that’s no bad thing. I think most people are doing this now. The other approach – that of self-modularising the exam is intriguing. I’m pretty sure it’s not what the designers of the exam had in mind, but you have to reluctantly admire the ingenuity (and deep pockets) of candidates who take this path.
Reason 4: The examiners are getting tougher.
I’ve examined with lots of colleagues and my impression is that they are not getting tougher and indeed many are acutely aware of how their trainees fear the exam. They are well briefed and trained and in my experience (which is limited) well prepared for the role. Amongst any group of examiners you will always find the hawks and doves, but there are mechanisms to avoid this affecting the results.
Examiners work in pairs and all results are overseen by the examinations team. Senior examiners ‘sit in’ on a proportion of examinations to give feedback to examiners on their style, questions, approach and marking. In other words there are mechanisms in place to avoid a significant examiner effect.
Reason 5: Content vs Time
The training program for EM is short. At just 5 years it is significantly shorter than that for specialities such as Paediatrics or Anaesthesia for example.
At the point of FCEM a trainee may have spent just 2 1/2 years in a senior role within an emergency department, yet the questions asked in some sections of the exam are perhaps tricky to gain without time served on the frontline – and perhaps more importantly in the back office.
I am thinking mostly about the management aspect of the exam. Now I am a really big fan of getting management into our training. It is the aspect of training that challenges us most when we become consultants, but the content needs to reflect the consultant of today and not of 10 years ago.
When I took the exam I, and most of my colleagues, would be applying for a job in a department with 1-3 consultants. The level and type of management ability needed then was different as it was entirely possible that you might have to deal with some really difficult problem ‘independently’ from day one. That’s not the case these days as most trainees will go to multi-consultant departments with excellent local support.
So, 10 years ago it would be quite reasonable to ask me to answer quite complex questions. These days, if someone asks a junior consultant how to deal with a trainee accused of a serious crime the answer could be to talk through all the components, but in reality they would never (Ed – should never) do this on their own. It is also unlikely that a trainee who has whizzed through training would have met all of the management topics in real life during their training. In my opinion we can and should exam trainees for a role that they are about to enter rather than one that they might take on as clinical director in years to come.
The exam has recently started including more ‘live’ management tasks that reflect shop floor management problems. This is a good thing.
Reason 6: The training isn’t very good
Tricky for me to judge this one. Training certainly varies around the country and the last time I saw data (about 2 years ago) it looked as though there were significant differences around the country. It is also a truth that EM docs are under a great deal of service pressure at the moment, this leads to a cycle that can diminish the opportunity and the quality of training so perhaps this increasing pressure in recent years does mirror the decline in pass rates. One of my great training colleagues has recently moved to Australia and he will be sadly missed here in Virchester. The trainees in Orange NSW will no doubt benefit whilst those here will lose out – and there are many similar examples across the country.
It’s also a truth that assessment drives learning. Trainees are really focused on the exam, but training in the workplace can only deliver exam preparation if the exam reflects the workplace. That link is vital and needs regular updating as our work and responsibilities change.[/learn_more] [learn_more caption=”Reason 7: Do the math….”] What exactly has maths got to do with a pass rate you may ask? Well, it’s all about probability. The exam has 5 components and ALL need to be passed INDEPENDENTLY.
This has a huge effect on the chances of a candidate passing on their first attempt. Let’s do a few sums with a few assumptions.
- 1. The examiners are pretty generous and good. They pass 80% of candidates in each element of the exam.
- 2. Candidates perform independently in each area of the exam. This is unlikely I know for reasons above and natural ability tends to spread across the board but stick with me.
- 3. All components are taken in the same year.
This is just a model for demonstration, and it is certainly the case the point 2 is a pretty big assumption for this, but as a method of design how might this multiple component model influence pass rates? So – ask yourself what’s the chance of a candidate passing first time with this model?
It’s a probability question and the answer is to multiply the probability of each element: so it’s (0.8)*(0.8)*(0.8)*(0.8)*(0.8). This comes out at a probability of 32.7%. Yes, that’s less than a third. So, based purely on probability as a result of taking multiple elements of the same exam (without compensation between elements) we reduce an overall pass rate of 80% (within each component) to a final pass rate of less than a third by dividing the exam into multiple parts.
In other words, every time we add a new independent element of the exam we require our trainees to be that little bit luckier to get through first time.
This is the part that really worried me as it reduces the overall chance of passing a very stressful and very expensive exam purely on a mathematical principle.
Interestingly, the first time pass rate for FCEM candidates is about 1/3 so perhaps the assumptions above are close to the mark. It also perhaps explains why there is a disparity between trainees and examiners. As an examiner I think that I mark generously and fairly. The numbers suggest that from an examiners perspective (in each element) they may pass the majority and that they believe that they are being generous. In contrast a trainee may feel that examiners are really tough – because they only have to slip up once in 5 attempts to fail.
Failure at the exam causes huge amounts of stress, demoralises trainees and prevents recruitment and progression in the speciality.
Maths solutions and concerns
How might we address this mathematical conundrum?
Approach 1: Make it all one big exam. Yes, you could do this so that all elements of the exam come together to a final pass mark. That would remove the mathematics problem but there will no doubt be those who argue that an ED consultant needs to be good at everything and that we cannot have someone who is really good in some areas, but not so good in others. This is a ridiculous argument really as the exam only tests a proportion of the syllabus in each diet. We already sample knowledge against the curriculum at the moment. At the moment we divide up into 5 different areas, but it is just as valid to divide it up in to 2/3/4/5/and number of elements. A greater number of divisions might feel right, but it is ultimately unhelpful
Approach 2: Restructure the exam to reflect the 2013 and beyond EP. I think the time is ripe to really look hard at what we want to see from the exam. We need to ask the question of whether it really tests the attributes of a modern (new) emergency medicine consultant. We need to keep many of the excellent components but also think about where there may be duplication (critical appraisal and CTR for example) and where we might introduce new elements that are more reflective of clinical practice.
Approach 3: Compensation. This is akin to a global mark but where a marginal fail in one area can be compensated for by an excellent pass in another area. I like this idea as it reflects the fact that trainees are not all the same (consultants aren’t either) and such a method goes some way to alleviating the mathematical error that multiple exams cause. This was the method used when I took the exam, with a marginal fail being compensated for by a good pass elsewhere. It is quite conceivable (actually quite likely) that I may have failed the management, but done well in the academic sections thereby passing overall. In today’s exam I would fail with such results.[/learn_more][learn_more caption=”So is change coming?”] The good news is that Dr Kevin Reynard will be updating the College on changes to the exam this month. The CEM conference at Twickenham will reveal how the exam will change and how this will alter our approach to examination and training.
Kevin is an excellent chap and has a long history of excellence in medical education. I for one am really looking forward to seeing what the next few years have to offer.[/learn_more]
The FCEM exam is such an important part of our speciality that it has to be right. It is so many things to emergency physicians, it is the bottleneck to promotion, the standard to which we aspire, the focus of training, a barrier to recruitment and progression. You might ask why change has not already taken place to reflect the changing roles of a new EM consultant. I suspect that the answer is in the requirements of the GMC and college to approve any changes which cannot be done too often. The GMC has the final say in aspects of examinations, length of training etc. The process is carefully controlled and if changes are to be announced this month then this will be the end product of a great deal of work by the college.
I really don’t know what they are planning but would love to see….
- An exam that covers all aspects of the syllabus – so must include management, academic and clinical components (though not all need be completed in a summative final assessment).
- An understanding that multiple independent components will lead to increased failure rates.
It is vitally important to protect the speciality and our patients and that’s why it has to be fair, equitable and not reliant on luck……, and for all of you who took the exam this week, and in the future, the very best of luck from St.Emlyn’s.
http://www.stemlynsblog.org/wp-content/uploads/2012/07/Simon-Carley.jpeg
References
- http://www.gmc-uk.org/Exam_Data_Summary_2010_2011.pdf_49438698.pdf
- http://www.gmc-uk.org/static/documents/content/CEM.pdf
A final word…..
Oh, and a request for a pause before commenting. Although we say that all comments are welcome at St.Emlyn’s I might suggest a pause before anyone suggests that the failure rate is a means of increasing funds to the college. I heard this once and was rather disgusted that anyone would think that the College (staffed by the excellent people that serve it) would even countenance such a thought.
Please don’t embarrass yourself by suggesting it.
I enjoyed the post and the statistics re fail rates in an exam with multiple independent elements are simple but compelling. Your reasons are valid but I have a little issue with No.2. The element of competition for entry into HST is missing in large parts of the UK. Competition drives up standards. When deaneries are looking at rotations without StRs to fill them with, candidates who meet a notional minimum standard will gain a place now where 8-10 years ago they would not have because the competition would have ensured selection of candidates well above the minimum standard. This might partially explain the timing in the performance drop-off from 2008 until present. In the East (which may or may not be comparable to Virchester), there have been more places than applicants since 2005 and exam performance drops off at the appropriate time right when the ST5s would be sitting for the first time in 2009.
I don’t think it’s a trainee vs non-trainee issue; all of our medics whatever their “grade” get equal access to study leave, education & training etc. The simple matter is for a multitude of reasons stretching back a decade we have not attracted high numbers of high quality candidates into EM HST and we should not be surprised when lower quality candidates struggle in the exam. I have no quick solution for this. Quality of training and work/life balance must be seen to be improving urgently to not deter our excellent ACCS trainees from an EM career and push them into a land of gas and gadgets. The exam is tough and so it should be (though the management syllabus in no way prepared me for the last 4 & 1/2 years of HR/GMC/OH stuff on my desk!) – I think evolution rather than revolution in the FCEM format is the way forward.
DOI – I passed first time in 2008.
Cheers David. That’s interesting about selection processes as it suggests that the appointment process has lowered standards to get people in. Is that really the case – if so it’s a worry….
I do agree that evolution rather revolution will be the way forward. The post is very conservative in that regard, at some point I might share a radical approach to exams, but I thought better of that until we see what the CEM has to offer.
As for the management then it is true that the exam probably cannot prepare for all the things you describe, but it does go someway to doing so. Colleagues in other specialities get little or none of this and many trusts are enforcing ‘new consultant’ courses to get people up to speed covering many of the topics we go through as part of EM training.
So yes, we do need to retain the elements that underpin the curriculum and the speciality (management, academic, clinical), but how that will look, and in particular how much is retained in the final summative exam will be interesting. I also wonder if some of the duplication between what is examined at MCEM and FCEM might diminish.
I’m looking forward to CEM conference to find out. Hope to see you there.
S
Thanks, Simon – on reflection there probably should be more management training, not less! Mrs Dave (Consultant radiologist) regularly consults your correspondent about management issues having had no training pre CCT whatsoever and I think is genuinely surprised at my level of expertise (given that she’s cleverer me in most other things) about complaints, the regulator, job planning, writing good e-mail etc.
Totally agree about reducing duplication.
Can’t make conference this year but I’ll keep checking Twitter for updates.
Dave
Cheers Dave.
Would love to meet up at some point. Maybe we can persuade you to come to SMACC?
http://www.smacc.net.au/
S
So let’s get my DOIs out of the way to start with: I’ve passed 3 of the subexams above (MCEM B and C, and FCEM CTR) – not all at the first attempt. I’m too old to have done MCEM A, sneaking in with MRCP1.
I may or may not have passed the SAQ and critical appraisal, depending on how coherent I was on Thursday.
I have yet to meet anyone in the examinations or training process who is actively malign, or motivated by the College wine cellar.
However, I am not sure that the examiners and trainers I have encountered are aiming for the same point. Anyone training in the Greater Virchester area will recognise “we’re training you to be good Emergency Physicians, not to pass the exam” from the TPD. This misses the point that in order to be allowed to be good EPs WE HAVE TO PASS THE BLOODY EXAM!!! (Sorry).
I have done lots of preparation for the SAQ, based on feedback from previous papers and recently successful candidates – I am uncomfortable about how much time I spent learning lists of tenuous relevance to safe practice – and yes, some of them did come up in the exam……to pick on a particular question (and I’m sure the College will reply that if it’s a bad question it will be moderated out): why do I need to know 4 specific non-ABC treatments for chlorine poisoning from memory? It would be more relevant to my being a safe/good consultant to ask me to demonstrate an ability to access Toxbase. At the same time, there wasn’t a question that addressed sepsis in adults, ACS or stroke – all (I would suggest) core knowledge for a competent practitioner.
So I agree with Simon’s concerns that the exam may not reflect what is expected of a new consultant, but my issues would be more around the clinical part of this than the management (of course I may change my tune after the management viva next month) 😉
Hi KC. Thanks for the thoughts.
I think what we would agree on is that the (1)training should reflect the (2)job which should reflect the (3)exam. In fact you can order those three any way you like and I think it still works…. they should be symbiotic, dependent and reflective of each other in a three way mix.
If we get those three in alignment it would (in the words of Matt from Big Blue Surfing) be ‘Happy Days’.
You also illustrate that any exam can only sample from the curriculum, testing a small part of what is in it, and an even smaller part of what is in it that can be tested in an exam (think about it….). This is a good argument for avoiding multiple exclusive components when designing an exam.
I am hopeful (as ever).
S
Oh, and a quick plug of your amazing revision notes that you put out onto your twitter feed. Yours, together with Natalie’s (who did similar) are #FOAMed gold.
Would be great to get them collated into one spot for future trainees.
S
Oh and Kirsty… Your notes really were amazing, you are a true artist in more than one way, thank you!
So… I am essentially an old style SpR in a new ST4/5/6 world. So far I have not been successful at ALL with the clinical section of the FCEM. As a result I have a bit of experience with sitting the beloved exam – the ceiling lights in the waiting area are now etched into my retinas. I think the fcem is always going to be a hard exam to get right, the examiners want to be sure that we as potential consultants know more than the ‘hot’ guidelines but I will admit the cyanide and chlorine questions confuse me as, are they really a good test of our abilities to treat patients and train juniors; or perhaps, out of London, exposure to such ailments are common?
My big issue however is when we are asked for essentially the same thing on multiple questions, in march it was difficult airways x2, on Thursday it was calculate paediatric fluid balance x3… Now I accept this is something I would expect people to know but are you just trying to humiliate those who are ignorant or the ones who are simply just bad at maths? Either way it seemed a little pointless to calculate it again and again!
I think too much emphasis is put on the numbers failing, does it really matter in the grand scheme of things? Why do I fail again and again I hear you ask, well frankly it probably comes down to two children, anti-social hours and possibly (last year) a bit of olympic obsession. So does this make me a bad doctor? I hope not! Do I think the training is too short? Probably…
What I do believe is that ultimately no exam is perfect and by failing so often it has shown me how MUCH my consultants care about me as a person, consults who I have never worked for and have barely met are offering help… and finally it has introduced me to a new generation of colleagues, ones who set up Facebook pages to help each other, people who go out of the way to share information and past papers, people who stand by each other when things get tough… So perhaps it’s not such a bad thing to fail after all!
Hi Lucy
The notes were entirely for my benefit – if they have helped others too that’s a welcome side-effect. I’m glad I’m not the only person who trudged through Thursday and thought “haven’t I already done this question”…..
I think comments about chlorine may be a recurring theme in the next few weeks(!). BTW, toxbase doesn’t have 4 non-ABC specific treatments so if any examiners would like to share the “correct” answer I’ll try to turn it into a mnemonic for next time 😉
K
Sorry to hear that Lucy.
The question about calculations is an interesting one, in fact the question of memory itself in modern day exams is a challenge in a age when we have instant access to the internet. For example I routinely check all drug doses in kids (even though I’ve prescribed for years) because I can and because it’s safe. Would we/should we/do we test memory in exams in a way in which we would not expect in day to day practice?
The consultants I know and work with are as you describe. My eldest starts ‘big school’ tomorrow and that feeling of pride and trepidation I feel tonight is oddly not that different to when our trainees go to FCEM. We really want them to achieve and really want to ensure that they are as well prepared as possible (that’s why many people become examiners). If a trainee fails from our department there is always a lot of soul searching and reflection. What could we do to make it better next time…..?
vb and the best of luck for this sitting.
S
PS – If you want to name some of those consultants who supported you feel free to do it here. St.Emlyn’s loves positive feedback 🙂
Interesting comments re memory. In this world of checklist medicine (and what looks like good evidence that we should accept and encourage checklists as good for patient safety) what message does it send out that we then examine exactly the process we have demonstrated doesn’t work in clinical practice? Outwith FCEM/APLS I wouldn’t dream of trusting my memory and mental arithmetic for paediatric dosing.
Food for thought.
Indeed. I’ve just got round to reading the checklist manifesto and I think it’s fair to say that the life support courses, MIMMS, HMIMMS etc. are really a form of checklist.
S
The changes to the exam will be out for formal consultation shortly as part of our curriculum changes going to the GMC. We will be proposing changes that reduce the overall burden of examinations throughout training as part of the submission. As part of this we will be removing some components, devolving others to training programmes and removing some limitations on others. The changes have been well received by trainers and trainees on the relevant College committees.
Unusually I disagree with the Virchester academics re compensation. I think the most valid and reliable component of fcem is the clinical part, accepting the limitations of what you can examine from the curriculum. As such, I expect our patients would not be happy being cared for by a Consultant who was unable to demonstrate the expected level of knowledge, skills and behaviours clinically- but was really good at writing a CTR. The decision to remove compensation was taken some years ago, supported by experts in medical education. In my time as Dean we will not be looking at this again.
Lastly, Simon has some very kind words to say about me. I have been very lucky to stand on the shoulders of giants throughout my career, and also bask in reflected glory from very talented trainees (starting at my time working at a hospital near Virchester). The academic drive, bright ideas and intellectual rigour behind these proposals have come from Ruth Brown (director of exams and academic VP) and Mike Clancy (ex Pres and Dean). My role has been largely decorative.
Please to respond to the consultation process- I hope people approve of the proposed changes- but if not we need to know!
Fantastic Kevin, that’s great to hear and I absolutely endorse your request to get involved.
In terms of compensation – actually I do agree with you. If I remember rightly in the old days the one area that could not be compensated for was the clinical. For all the reasons you describe that is right and proper. My apologies for missing that key point out in the blog post.
vb
S
It’s very interesting… On this side of the pond, a similar phenomenon is going on… The American’s have recently seen a drop in the pass rate of the ABIM exams. I wrote a reactionary post debating the merits of purely blaming the learners….
Is the training matching the exams? Should they? Your raise the issue of ‘gamesmanship’ and it’s true… exams can force behaviours and shift educational programs to respond. Assessment drives learning because it tells people what to value.
Check out my post at: http://boringem.org/2013/06/30/are-we-training-dumber-doctors-counterpoint-from-a-millenial/
Eric Holmboe calls it “Nostalgia-itis imperfecta” to say that it’s all the learners that are to blame… To say we were better way back in the day… etc..
T
The College proposals are out for consultation- please go to the college website and read the doc and then do the appropriate survey. They also include WPBA proposals
Thank you