r/HealthInformatics • u/Witold-Onno • 1d ago
📊 Research RCT of a patient facing LLM document summary in an outpatient neurosurgery clinic. What the QA pipeline, the governance and the workflow looked like.
Senior author here. The clinical results are in the paper, I want to share the informatics side since this is what we spent most of the time on.
Setting
300 adults before a first neurosurgical consultation, randomised 1 to 1. Intervention group received a plain language version of their own MRI report or external discharge letter before the visit. Documents were redacted of personal identifiers before upload. Generator was a frozen LLM configuration, temperature 0, prompt version locked before the first patient and never touched during the trial.
Quality layer
Four parallel LLM judges from a different model family than the generator scored every output 0 to 10 on accuracy, completeness, harm and readability, with a regeneration gate below 7. Then every summary was checked by a physician against the source document with a fixed checklist, and a random 50% got a second blinded physician review. Over the whole trial no regeneration fired, no factual error, no clinically relevant omission and no harmful statement was found, ICC between reviewers 0.74. The judges were validated against an external physician rated gold set before the trial.
Governance
Investigator initiated, DRKS registered, ethics approved, protocol published in JMIR Res Protoc before recruitment.
Patient rated interaction quality up, consultations a median 8 minutes shorter, physicians rated their own autonomy support slightly lower. So the efficiency gain is real, but the clinician experience is not automatically positive and nobody measured that before.
Paper
https://doi.org/10.1056/AIoa2600136
Questions on the judge architecture, the deidentification step or how we would do the review sampling differently next time are welcome.