In October, I had the opportunity to give a talk on the ethics of Artificial Intelligence at the EUSEM Congress in Copenhagen. My focus was on the ethical implications of AI for emergency care, a topic that’s becoming increasingly relevant as advances in AI accelerate. Below is a rundown of some of the key points I covered, touching on where we are now, where we might be headed, and the ethical questions we can’t ignore along the way.
Artificial Intelligence in Emergency Care: Too Good to Be True?
AI has been on our radar in emergency medicine for years, promising to streamline workflows, enhance diagnostic accuracy, and help decision-making. But AI in healthcare isn’t new, and we’ve had some high-profile bumps along the way. Take Babylon Health, for example. Initially deployed within the NHS, their chatbot claimed it could assess symptoms and provide advice, replacing the need for direct human triage in some cases.
But when I tried entering symptoms that were typical for a heart attack, the chatbot suggested I was having a panic attack. Stories like this raise doubts about reliability and highlight how tricky it is to build trust in healthcare AI. Since then, AI has come a long way, with tools like ChatGPT offering far more advanced capabilities. When I was preparing for my talk at EUSEM, I asked ChatGPT to role-play with me and imagine that I was a patient with chest pain in the ED. ChatGPT issued comprehensive recommendations for my care, including recording an ECG, measuring troponin, indications for serial sampling, the use of the HEART score [Ed – I might question why it didn’t advocate for T-MACS given that T-MACS has performed better in direct comparisons, but that’s another matter!]. I’d even told ChatGPT that I smoke [Ed – I don’t, it was just for the sake of role play!], and it emphasised the importance of smoking cessation advice – something that I think emergency physicians often forget when focused on the acute situation! It felt like it may not be long before large language models (LLMs) like ChatGPT can help guide our healthcare.
New Horizons in AI: Radiology, Staffing, Surgery, and Clinical Intelligence
From radiology reporting to dynamic staffing, robotic surgery, and ambient clinical intelligence, the new generation of AI tools promises to shake up healthcare as we know it.
- Radiology Reporting: AI is now used to analyze medical images, flagging abnormalities for emergency physicians and/or radiologists to review. This doesn’t replace the radiologist’s job but can make it faster and more accurate by prioritizing critical cases.
- Dynamic Staffing: AI-driven scheduling tools predict demand and adjust staffing accordingly. By forecasting patient flow, these tools could prevent overcrowding and make EDs more efficient. In future, perhaps AI will be able to write rotas and even identify future staffing needs, guiding recruitment.
- Robotic Surgery: Robots aren’t just performing surgery but are learning to refine techniques based on their past performance. Their precision has huge potential, although the learning curve and trust barrier remain high.
- Ambient Clinical Intelligence: Imagine AI assistants documenting patient encounters in real-time, freeing clinicians from their screens and allowing them to focus on patients. Tools like these are designed to help reduce burnout and improve the quality of interactions in healthcare settings. I recently learned that Chris Baugh and his team at Brigham and Women’s Hospital in Boston, US, are using an app for ambient documentation. They consult with a patient while the app is recording, and the app will write up the clinical notes. Clearly, there will be some issues to iron out along the way (including the concern about possible hallucinations by LLMs!), but this technology could really revolutionise the way that we work in the ED.
The Double-Edged Sword of AI in Healthcare
AI’s advantages are significant—potentially greater accuracy, efficiency, and even improved patient safety. But it comes with challenges, too, from the “black box” nature of some models to regulatory and monitoring issues, and the ever-present problem of bias.
Let’s unpack a few:
- The Black Box Effect: Many AI models don’t offer a clear rationale for their recommendations. In clinical settings, this can make it difficult to justify decisions based on AI alone, especially when outcomes are high-stakes.
- Monitoring and Regulation: Unlike a human clinician, AI isn’t bound by professional or ethical codes. How do we create oversight that keeps up with technology’s pace?
- Bias in AI: AI systems are trained on vast datasets, but if these datasets don’t represent certain demographics accurately, AI could perpetuate or even amplify these biases. For example, people from underrepresented backgrounds, whose data may be missing or unrepresentative, risk receiving suboptimal recommendations.
Thought Experiment #1: Would You Trust an AI Doctor?
Imagine it’s 2035, and you’re experiencing chest pain. You have two options: visit the ED as usual or consult a highly accurate AI health assistant, which large studies have shown is as accurate as human doctors. Which would you choose?
Trust in AI remains a huge hurdle. In my talk, I explained that LLMs, like ChatGPT, function by calculating the probabilities of word sequences based on vast datasets—essentially, they predict what makes the most sense given the patterns they’ve learned. They don’t critically evaluate data or verify sources; they just “speak” based on statistical likelihood. I explained the concept of “temperature” in LLMs, which controls how creative or “confident” an AI will be. A lower temperature produces more predictable responses, while a higher temperature leads to more diverse, and potentially risky, output.
This brings us back to trust: Would we feel comfortable handing over critical health decisions to something that’s fundamentally based on probability, not understanding? Remember, an LLM will not critically appraise the content within its dataset: it returns outputs based on how often it encounters something, not based on reasoning, as a human would.
Bias in AI: The Problems We Face
AI can mirror our biases, often in ways we’re not even aware of. This happens when training data isn’t fully representative or when certain groups are left out of data collection. If certain communities are underrepresented in data, AI may make less accurate recommendations for them, unintentionally reinforcing inequities.
For instance, an AI trained mostly on Western datasets might not accurately reflect the health needs or symptoms of individuals from different ethnic backgrounds, age groups, or those with rare health conditions. And as we increase our reliance on AI in healthcare, these biases can get woven deeper into the system, affecting diagnosis, treatment recommendations, and ultimately, patient outcomes.
Thought Experiment #2: Human ED or Robot ED?
Fast-forward to 2060. Now, you have a choice between two EDs: the traditional human-staffed ED or a fully automated, robot-run ED, which is faster, safer, and more accurate. Which one would you pick?
Choosing a robot ED might appeal because of efficiency and reduced waiting times, but what about the “human touch” that many of us still crave? Emergency care isn’t just about treating symptoms; it’s about understanding fears, offering reassurance, and building trust. Not to mention that in a robot-run ED, those with limited digital literacy or without access to digital devices might face even more health inequalities.
Thought Experiment #3: The Health Guardian App
Now, imagine an app called Health Guardian, powered by AI and constantly analyzing data from your digital footprint—wearables, conversations, shopping records, health records—to predict health risks and provide preventive care recommendations. Would you use it, and what would you be willing to share to optimize your health?
While this could transform healthcare by preventing illness before it starts, it also raises profound privacy concerns. Health Guardian would know intimate details about you, and many people may feel uncomfortable with that level of surveillance—even if it promises better health outcomes. Some of the concerns that people may have include the impact on insurance – life insurance, travel insurance, car insurance, etc.; and the potential impact of a breach of security. What would a data leak mean for people’s privacy?
Final Thoughts: The Best or the Worst Thing to Happen to Humanity?
AI could be revolutionary for emergency care, making it more accurate, efficient, and possibly even more equitable. But with the promises come ethical quandaries, technical challenges, and the risk of losing the very human touch that is central to medicine.
As Stephen Hawking said, “The rise of powerful AI will be either the best or the worst thing ever to happen to humanity. We do not yet know which.” It’s up to us to steer this technology responsibly, balancing innovation with compassion and ensuring that AI truly serves our needs—not the other way around.
Acknowledgements: I would like to thank Liz Crowe and the entire St Emlyn’s team for their invaluable input in drafting this blog post.
