Quick takeaways

  • Drafting a note is a much safer assignment than choosing a diagnosis or treatment.
  • A passing medical-exam score does not show that a model can handle a messy patient visit.
  • I would let an LLM prepare work for a doctor. I would not let it own the decision.

Ask a doctor what part of the job follows them home and there is a good chance the answer involves notes. After the appointment comes the documentation: symptoms, exam findings, orders, reasoning, follow-up. None of it is optional. It is also hard to give a patient your full attention when you know a screen full of unfinished charts is waiting.

This is the first hospital job I would hand to a large language model. An ambient scribe listens during the visit and turns what it hears into a draft note. The doctor checks it, changes it and signs it. Nuance, Suki and Epic all have versions of this idea. Less typing after clinic can leave more attention for the patient during the visit.

The word draft sets the limit. The model has not examined the patient and cannot tell whether a pause meant confusion or a symptom sounded more serious in the room. Its output should begin the paperwork and remain subject to a person's review.

Exam scores leave out the patient

Google built Med-PaLM 2 for medical question answering. In the Nature paper, it reached 86.5 percent accuracy on questions modeled after the U.S. Medical Licensing Examination (Singhal et al.). GPT-4 also performed well on medical challenge problems (Nori et al.). The results show that a general language model can answer many difficult medical questions.

I still would not confuse that with treating a patient. A test question is designed to contain the clues needed for one best answer. A real patient may forget a medication, describe pain strangely or have two conditions at once. The doctor also has to notice what was never typed into the prompt.

That gap is where benchmark excitement can get misleading. An LLM can produce a polished explanation with the right diagnosis in it. It can produce the same polished explanation with the wrong diagnosis too. Fluency makes the error harder to spot because the answer does not look uncertain.

The assignment changes the risk

I see a useful dividing line. Let the model retrieve a guideline, organize a long chart or suggest questions the doctor may want to ask. Those jobs make information easier to handle. The physician still decides what the information means for this patient.

A safer division of work The model can retrieve, organize, suggest and draft; the clinician keeps the final diagnosis, doses, treatment choice and patient explanation. A safer division of work LET THE MODEL retrieve a guideline · organize a chart · suggest questions · draft a note KEEP WITH THE CLINICIAN final diagnosis · medication doses · treatment choice · explain it to the patient
Schematic: The model prepares information while the clinician owns the medical decision.

Medication doses and final diagnoses are different. A wrong summary wastes time. A wrong dose can hurt someone before the error is caught. Shah and colleagues argue that hospitals need deliberate evaluation and adoption rules for clinical LLMs, which sounds obvious until you remember how quickly ordinary software tools spread through a workplace (Shah et al.).

Another problem is automation bias. Once a suggestion appears neatly inside the chart, disagreeing with it takes effort. A rushed clinician may trust the model too much. Requiring a human click is weak protection if everyone learns to click through.

Where I draw the line

I am in favor of doctors using LLMs. These tools are already entering medicine, and banning a useful note draft because the model might make an error makes little sense when a doctor is reviewing it. The hospital should test the system on its own patients and keep measuring what it misses.

I would stop before the model becomes responsible for care. It can prepare, search and draft. The diagnosis, treatment choice and explanation to the patient still belong to a clinician who can be questioned and held accountable. I would keep that line visible in every clinical use.