In 2001: A Space Odyssey, Dave returns to the spacecraft to find himself locked out. The onboard AI, HAL, has decided Dave is a threat to the mission. Dave pleads, reasons and commands. But HAL has already made up its mind; and won’t be moved.
Last week I attended a patient with chest pain; anxious not because of his symptoms, but because ChatGPT told him he was having a heart attack. He typed in his symptoms, received a confident response, and now sat in our ambulatory area convinced that anything I said to reassure him was either incompetence or dismissiveness. ChatGPT had made up his mind; and now he won’t be moved.
His ECG was non-diagnostic. His troponin was negative. His pain was non-cardiac. “But why has ChatGPT been so certain !?”
This scenario is becoming more familiar. Patients arrive having consulted AI before consulting us – sometimes openly, sometimes revealed only at the end of the consultation, sometimes never disclosed at all. What strikes me isn’t the information-seeking itself (we’ve navigated “Dr Google” for two decades), but something new: the authority patients ascribe to these readily available AI tools. And here lies the problem:
For many patients AI isn’t seen as the starting point of this conversation. It is seen as the expert against which we as clinicians are measured.
What the research tells us
AI language models (such as ChatGPT) get atypical presentations wrong. It makes hasty diagnoses and doesn’t adhere to guidelines. When tested on cases without typical features – pneumonia without fever, for instance – all AI models failed. This matters for emergency medicine, where atypical presentations are our daily reality.
AI language models miss emergencies. Studies examining various AI models’ ability to recognise emergencies found they produced false negatives – failing to identify some genuine emergencies while simultaneously over-triaging minor presentations.
Patients can’t tell good AI advice from bad. People couldn’t distinguish AI-generated from physician responses – and then preferred the AI response! Even low-accuracy responses were convincing. Its persuasive, fluent language creates false confidence.
AI language models sound more empathetic than we do. ChatGPT scored higher on empathy (4.18 vs 2.7) and usefulness (4.04 vs 2.98) compared to physician expert panels. The AI model feels more helpful – even when it may be less safe.
Why this is harder than “Dr Google”
We’ve managed internet-informed patients for twenty years. Research on “Dr Google” actually found mostly positive effects: clinicians could capitalise on patients’ ideas, concerns and expectations. But AI language models are different…
The confidence asymmetry. Google returns a list of possibilities and sources. AI language models deliver a single, confident-sounding answer in authoritative prose. The uncertainty that might prompt a patient to seek clarification is flattened.
The conversational frame. Patients interact with an AI model – they ask follow-up questions, and get personalised-seeming responses. This creates a relationship-like dynamic that search results don’t.
The empathy mimicry. In a time-pressured ED consultation, we can’t match the length and apparent attentiveness of an AI model’s response. The comparison disadvantages us.
The invisible limitations. AI doesn’t know what it doesn’t know. It can’t examine. It can’t integrate the subtle clinical signs that change everything. But it doesn’t communicate these limitations – it just answers confidently (also referred to as hallucination).
Sometimes the AI model is right. This is the uncomfortable bit. Sometimes the AI model raises something worth discussing, or prompts a patient to attend who might have stayed home.
The problem isn’t that patients seek information before seeing us. It’s that the AI language model’s false certainty forecloses a conversation we need to have.
This will change
I should be clear: AI language models are not there, yet. Current models can’t examine patients, can’t integrate real-time clinical findings, can’t recognise the subtle patterns that change everything. They’re trained on textbook presentations and struggle with the messy reality of emergency medicine.
But yet, is doing an awful lot of work in that sentence.
The NHS 10-Year Plan explicitly includes AI triage and digital-first care as part of the transformation agenda. This isn’t a consumer trend we’re reacting to – it’s the direction of travel for the health service itself. I’ve written previously about what successful AI integration might look like: AI that augments clinical judgment, rather than replacing it, backed by proper accountability frameworks. But we’re not there yet.
The gap between what AI can do and what we can do will narrow. That makes this moment important.
We’re not just managing a temporary problem until AI improves. We’re establishing patterns for how clinicians and AI-informed patients interact. The communication habits we build now will most likely shape this relationship for years to come.
What seems to work
Research identifies two main approaches clinicians take in these situations: defensive and participative.
The defensive approach – discouraging patients from using online sources – is largely ineffective.
The participative approach shows more promise. This includes understanding patients’ emotional needs, instructing on appropriate use, and jointly examining the information together. Studies found the participative approach encouraged patients to express concerns and continue asking questions. The conversation opens rather than closes.
There’s wisdom in a theme that emerges from research on internet-informed patients: never ridicule information a patient found online.
Emergency medicine-specific guidance on health misinformation is slowly emerging. An US expert panel noted that most physicians lack communications training to address misinformation – an obvious gap in emergency medicine training. They identified approaches including broaching the topic directly, debunking effectively, and “prebunking” (providing accurate information before exposure to misinformation).
Three ways to respond (and one to avoid)
- Invite disclosure early. Patients often conceal AI use expecting disapproval. “Have you looked anything up about this?” asked without judgment, creates space for honesty. You can’t address what isn’t disclosed.
- Acknowledge uncertainty, then contextualise. We miss things too. The patient’s instinct to seek a second opinion isn’t unreasonable. But we can do what AI can’t: weigh probability. “Your ChatGPT might be right. But with these findings, the probability of a heart attack is very low. Here’s what’s more likely, and here’s what would change my thinking.” You’re not dismissing them; you’re showing how clinical reasoning works.
- Handle the unconvinced patient explicitly. Sometimes you’ll do everything right and the patient still trusts the AI more. Name it: “I can see you’re not fully reassured. Let me explain what would need to change for me to be worried, and what you should watch for at home.” State your reasoning out loud: “I’m putting together what you’ve told me, what I’m seeing when I examine you, and what I know about how these presentations evolve.” Give them a framework, not just a conclusion. Make the invisible work visible.
A riskier option. Everything mentioned is about you controlling the conversation: inviting disclosure, engaging with the information, explaining your reasoning. But what about engaging directly with the AI model?
You could invite the patient to enter the diagnostic findings into their chat: “Tell ChatGPT your ECG was normal and your troponin was negative.” In theory, it should update its assessment. But AI language models can be unpredictable. They don’t reason like clinicians. They generate plausible text based on pattern-matching, not Bayesian reasoning.
With new data an AI model may reassure, hedge or introduce new concerns. You can’t control a narrative if you hand part of the conversation to something inconsistent.
The deeper challenge
The patient who trusts an AI language model more than you isn’t making an irrational choice. They’re responding to an information environment where a confident, accessible, always-available AI language model competes with a rushed, fragmented, clearly-stressed healthcare system. The AI gives them time and certainty, when we give them queues and complexity.
In my previous posts on self-organised criticality and design as repair, I’ve written about how EDs function at the edge of chaos and how invisible repair work keeps systems running. When dealing with patients empowered by AI, we’re not just managing clinical uncertainty, we’re managing epistemic competition in conditions that systematically disadvantage us. ChatGPT has infinite patience. We have ten minutes and a waiting room full of patients.
But it also suggests something more fundamental: we need to get better at engaging with misinformation under pressure, and the system needs to give us the conditions to do so. Neither is easy. The first requires training we mostly don’t receive (yet). The second requires structural change that is for now out of reach.
Have you encountered patients who’ve consulted an AI language model? How do you navigate these encounters?
