I was recently reviewing a number of documents relating to outcomes of training for one of my other jobs at St.Emlyn’s. There is a lot of interesting data out there about how well trainees perform in professional exams and also during their end of year training assessments (the ARCP).
Anway, most of this data is publicly available but perhaps not widely read as it can be a little challenging at times. However, I was particularly interested to look at data from the recent review of training in Emergency Medicine by the GMC and also from the web published Annual Speciality report from the GMC. Together with the Annual Speciality report from the College of Emergency Medicine they make for some really interesting reading regarding the future of training in the UK.
Why is this important?
It’s important because EM faces a recruitment crisis in the UK and it is possible that the examination/qualification route may be a reason for this. The current model of training requires our trainees to pass many exams.
MCEM – Divided into 3 parts. All must be passed
- Part A – Basic Sciences
- Part B – Short answer questions
- Part C – OSCE
FCEM – Divided into 5 parts. All parts must be passed. Critical appraisal component now taken earlier than other three parts. SAQ and OSCE must be passed at the same sitting.
- Critical Appraisal
- Short answer questions
- Management viva
- Critical Topic review
That’s a LOT of exams to be taken in a relatively short training program and it seems to me that my trainees are always studying for an exam from the moment they choose our speciality to the point at which they eventually pass FCEM. This is not a happy journey and anyone who works with trainees cannot help but feel that there might be a better way.
I’m interested in this after looking at educational outcomes across many specialities as part of my other roles in education. There is some fascinating stuff out there (if you’re into that sort of thing), and at St.Emlyn’s we obviously took a microscope to the educational outcomes in UK emergency medicine. We were surprised, and worried about what we found and thought we should share the data that’s out there, and also look to see how changes to the structure of the exam might improve the educational outcomes for our trainees…….but before we consider the future, let’s just have a think and a delve into the current situation and ask ourselves why we might need to change…..
- I’m an FCEM examiner. I did FCEM and passed first time when the exam was very different.
- I try to help trainees get ready for the exam and have written on the subject.
- I became an examiner when the consultants in our department complained that we did not understand why our trainees were failing the exam. Becoming an examiner has (I think) made me a better trainer and I would recommend all consultants to get involved. It takes time and effort but it’s very rewarding.
- I have received expenses (travel/hotel) as an examiner.
- I have helped write and review aspects of the exam in recent years.
- I believe that the current examiners and college are doing a good job in delivering the current exam system.
- The examiners I have worked with have been excellent. They are highly motivated and put a huge amount of effort into the exam. This is often in their own time and at some personal expense in terms of time and energy.
- The views expressed here are entirely my own thoughts and in no way represent a college/hospital/training program view.
Let’s look at pass rates.
There are two things that I want to know about pass rates. The first is how many people get through the exam and the second is how many get through at their first attempt.
The data for the first of these questions is out there. If we look at data from the Annual Speciality review and from recent GMC education review in EM we can get the following information.
Overall FCEM pass rates.
- 46.7% in 2011-12
- 48.9% in 10-11
- 50.8% in 09-10
- 60.3% in 08-09
This worries me a great deal as this means that at the time of sitting, which is by definition at a point where enough training has been completed to be eligible for the exam then fewer than half the candidates pass, and the results are getting worse. We are looking at a year on year decrease in pass rates and we need to think hard about why as this can only be demoralising for the candidates and in all honesty quite frustrating for trainers.
Overall FCEM pass rates.
46.7% in 2011-12
48.9 in 10-11
50.8 in 09-10
60.3 in 08-09http://t.co/CegfidHris
— Simon Carley (@EMManchester) August 13, 2013
Reason 1: Is the exam getting harder?
I don’t think so. The format has remained the same over the last four years and if anything it should perhaps have become easier.
Trainees and trainers should be more familiar with the format, will have received feedback from past FCEM’ers and examiners and there are only so many questions that can be recycled back and forth into the test.
Greater familiarity and an ironing out of ‘bad’ questions would arguably lead to an increase in pass rates so I don’t think this is the reason
Reason 2: The trainees are not as bright
Now. This could be a tricky one….are the trainees less clever than they were?
No, certainly not the ones I deal with anyway. The College dean has suggested that there are a greater numbers of doctors taking the exam from non-traditional training programs (at least that’s what I think he is suggesting), and that this may reduce the overall quality of candidates.I cannot comment because I don’t have that information available, but it is possible that the type and preparation of candidates from then and now is different and that this is reflected in the results.
If that is the case then I am quite perturbed. I don’t really like to divide our junior doctors into trainees and non-trainees. All doctors are trainees from the day they decided to do medicine and we do our non-deanery appointed colleagues a dis-service to think otherwise (I’m sure @CEMdean thinks same in practice)
Reason 3: Gamesmanship
The FCEM requires a candidate to pass all 5 parts of the exam, but not at the same time. Alth0ugh the clinical components need to be passed at the same sitting (OSCE and SAQ), the other elements can be passed at different times. That means that if you have the time (and the money) then you can approach the exam in a modular fashion. To some extent this already happens as candidates can take the critical appraisal component from year 4 onwards and that’s no bad thing. I think most people are doing this now. The other approach – that of self-modularising the exam is intriguing. I’m pretty sure it’s not what the designers of the exam had in mind, but you have to reluctantly admire the ingenuity (and deep pockets) of candidates who take this path.
Reason 4: The examiners are getting tougher.
I’ve examined with lots of colleagues and my impression is that they are not getting tougher and indeed many are acutely aware of how their trainees fear the exam. They are well briefed and trained and in my experience (which is limited) well prepared for the role. Amongst any group of examiners you will always find the hawks and doves, but there are mechanisms to avoid this affecting the results.
Examiners work in pairs and all results are overseen by the examinations team. Senior examiners ‘sit in’ on a proportion of examinations to give feedback to examiners on their style, questions, approach and marking. In other words there are mechanisms in place to avoid a significant examiner effect.
Reason 5: Content vs Time
The training program for EM is short. At just 5 years it is significantly shorter than that for specialities such as Paediatrics or Anaesthesia for example.
At the point of FCEM a trainee may have spent just 2 1/2 years in a senior role within an emergency department, yet the questions asked in some sections of the exam are perhaps tricky to gain without time served on the frontline – and perhaps more importantly in the back office.
I am thinking mostly about the management aspect of the exam. Now I am a really big fan of getting management into our training. It is the aspect of training that challenges us most when we become consultants, but the content needs to reflect the consultant of today and not of 10 years ago.
When I took the exam I, and most of my colleagues, would be applying for a job in a department with 1-3 consultants. The level and type of management ability needed then was different as it was entirely possible that you might have to deal with some really difficult problem ‘independently’ from day one. That’s not the case these days as most trainees will go to multi-consultant departments with excellent local support.
So, 10 years ago it would be quite reasonable to ask me to answer quite complex questions. These days, if someone asks a junior consultant how to deal with a trainee accused of a serious crime the answer could be to talk through all the components, but in reality they would never (Ed – should never) do this on their own. It is also unlikely that a trainee who has whizzed through training would have met all of the management topics in real life during their training. In my opinion we can and should exam trainees for a role that they are about to enter rather than one that they might take on as clinical director in years to come.
The exam has recently started including more ‘live’ management tasks that reflect shop floor management problems. This is a good thing.
Reason 6: The training isn’t very good
Tricky for me to judge this one. Training certainly varies around the country and the last time I saw data (about 2 years ago) it looked as though there were significant differences around the country. It is also a truth that EM docs are under a great deal of service pressure at the moment, this leads to a cycle that can diminish the opportunity and the quality of training so perhaps this increasing pressure in recent years does mirror the decline in pass rates. One of my great training colleagues has recently moved to Australia and he will be sadly missed here in Virchester. The trainees in Orange NSW will no doubt benefit whilst those here will lose out – and there are many similar examples across the country.
It’s also a truth that assessment drives learning. Trainees are really focused on the exam, but training in the workplace can only deliver exam preparation if the exam reflects the workplace. That link is vital and needs regular updating as our work and responsibilities change.[/learn_more] [learn_more caption=”Reason 7: Do the math….”] What exactly has maths got to do with a pass rate you may ask? Well, it’s all about probability. The exam has 5 components and ALL need to be passed INDEPENDENTLY.
This has a huge effect on the chances of a candidate passing on their first attempt. Let’s do a few sums with a few assumptions.
- 1. The examiners are pretty generous and good. They pass 80% of candidates in each element of the exam.
- 2. Candidates perform independently in each area of the exam. This is unlikely I know for reasons above and natural ability tends to spread across the board but stick with me.
- 3. All components are taken in the same year.
This is just a model for demonstration, and it is certainly the case the point 2 is a pretty big assumption for this, but as a method of design how might this multiple component model influence pass rates? So – ask yourself what’s the chance of a candidate passing first time with this model?
It’s a probability question and the answer is to multiply the probability of each element: so it’s (0.8)*(0.8)*(0.8)*(0.8)*(0.8). This comes out at a probability of 32.7%. Yes, that’s less than a third. So, based purely on probability as a result of taking multiple elements of the same exam (without compensation between elements) we reduce an overall pass rate of 80% (within each component) to a final pass rate of less than a third by dividing the exam into multiple parts.
In other words, every time we add a new independent element of the exam we require our trainees to be that little bit luckier to get through first time.
This is the part that really worried me as it reduces the overall chance of passing a very stressful and very expensive exam purely on a mathematical principle.
Interestingly, the first time pass rate for FCEM candidates is about 1/3 so perhaps the assumptions above are close to the mark. It also perhaps explains why there is a disparity between trainees and examiners. As an examiner I think that I mark generously and fairly. The numbers suggest that from an examiners perspective (in each element) they may pass the majority and that they believe that they are being generous. In contrast a trainee may feel that examiners are really tough – because they only have to slip up once in 5 attempts to fail.
Failure at the exam causes huge amounts of stress, demoralises trainees and prevents recruitment and progression in the speciality.
Maths solutions and concerns
How might we address this mathematical conundrum?
Approach 1: Make it all one big exam. Yes, you could do this so that all elements of the exam come together to a final pass mark. That would remove the mathematics problem but there will no doubt be those who argue that an ED consultant needs to be good at everything and that we cannot have someone who is really good in some areas, but not so good in others. This is a ridiculous argument really as the exam only tests a proportion of the syllabus in each diet. We already sample knowledge against the curriculum at the moment. At the moment we divide up into 5 different areas, but it is just as valid to divide it up in to 2/3/4/5/and number of elements. A greater number of divisions might feel right, but it is ultimately unhelpful
Approach 2: Restructure the exam to reflect the 2013 and beyond EP. I think the time is ripe to really look hard at what we want to see from the exam. We need to ask the question of whether it really tests the attributes of a modern (new) emergency medicine consultant. We need to keep many of the excellent components but also think about where there may be duplication (critical appraisal and CTR for example) and where we might introduce new elements that are more reflective of clinical practice.
Approach 3: Compensation. This is akin to a global mark but where a marginal fail in one area can be compensated for by an excellent pass in another area. I like this idea as it reflects the fact that trainees are not all the same (consultants aren’t either) and such a method goes some way to alleviating the mathematical error that multiple exams cause. This was the method used when I took the exam, with a marginal fail being compensated for by a good pass elsewhere. It is quite conceivable (actually quite likely) that I may have failed the management, but done well in the academic sections thereby passing overall. In today’s exam I would fail with such results.[/learn_more][learn_more caption=”So is change coming?”] The good news is that Dr Kevin Reynard will be updating the College on changes to the exam this month. The CEM conference at Twickenham will reveal how the exam will change and how this will alter our approach to examination and training.
Kevin is an excellent chap and has a long history of excellence in medical education. I for one am really looking forward to seeing what the next few years have to offer.[/learn_more]
The FCEM exam is such an important part of our speciality that it has to be right. It is so many things to emergency physicians, it is the bottleneck to promotion, the standard to which we aspire, the focus of training, a barrier to recruitment and progression. You might ask why change has not already taken place to reflect the changing roles of a new EM consultant. I suspect that the answer is in the requirements of the GMC and college to approve any changes which cannot be done too often. The GMC has the final say in aspects of examinations, length of training etc. The process is carefully controlled and if changes are to be announced this month then this will be the end product of a great deal of work by the college.
I really don’t know what they are planning but would love to see….
- An exam that covers all aspects of the syllabus – so must include management, academic and clinical components (though not all need be completed in a summative final assessment).
- An understanding that multiple independent components will lead to increased failure rates.
It is vitally important to protect the speciality and our patients and that’s why it has to be fair, equitable and not reliant on luck……, and for all of you who took the exam this week, and in the future, the very best of luck from St.Emlyn’s.
A final word…..
Oh, and a request for a pause before commenting. Although we say that all comments are welcome at St.Emlyn’s I might suggest a pause before anyone suggests that the failure rate is a means of increasing funds to the college. I heard this once and was rather disgusted that anyone would think that the College (staffed by the excellent people that serve it) would even countenance such a thought.
Please don’t embarrass yourself by suggesting it.