Route, don't write
AI that drafts replies to patients has not reliably saved doctors time. Health systems should first use AI to sort the inbox, and draft clinical replies only where audits show a gain.
Listen to this special topic
read by an AI voiceAI drafts of replies to patients take doctors longest to correct when the message most needs a doctor's judgment. A study published at the end of August examined how more than a thousand physicians at UC San Diego Health edited replies drafted for them by software in the health system's Epic record. Each reply was timed from the moment a doctor began it until it was sent, and every kind of change the doctors made was associated with a slower reply. After allowing for how complex the message was, the largest increases in time came with changes that explained a scan or a laboratory result, or that clarified or ruled out a diagnosis.
The commonest edit, nearly two in five by the study's count, involved booking or moving an appointment, and because such administrative edits were so frequent, they added the most work across the health system. The first job for AI in the inbox should be routing: sending requests for appointments, refills and forms to the staff who handle them, and lifting urgent symptoms to the top. Drafts of clinical replies should be switched on only where a local audit shows they save time without adding errors.
Earlier studies in clinics found little or no saving of time. A randomized quality improvement study at UC San Diego, published in 2024, found that the drafts did not shorten the time primary care doctors spent replying and came with longer reading times and longer replies. At Stanford, drafts written for primary care and gastroenterology clinicians during a five-week pilot in 2023 did not save them time, though in surveys before and after the pilot, answered by fewer than half of them, the clinicians reported a lighter mental load and less exhaustion. Where drafts did save time, the gain was small: at Children's Hospital of Philadelphia, clinicians saved about three seconds a message. Drafts also make mistakes. In an experiment at Mass General Brigham, OpenAI's GPT-4 invented 100 questions from patients with cancer and drafted replies to them; radiation oncologists answered the questions themselves and then edited the drafts. In about one in 14 of their assessments, the doctors judged that a draft sent unedited risked severe harm, and in one, death.
Routing has been studied on a far larger scale. In 2022 Kaiser Permanente in Northern California began using software that labels patients' messages by topic and sorts them to the right staff. In five months of 2023, its regional team resolved nearly a third of almost 5 million messages before they reached the physician they were addressed to, most of them through medical assistants and telephone staff. The study, by the group's own researchers, was a quality improvement report.
Kaiser's Southern California medical group built a tool that categorizes messages by topic and flags urgent cases as top priorities. In clinics that adopted it, the median wait before a clinician first read a message the tool marked urgent fell from about 22 hours to 5; in similar clinics without the tool, it fell from about 18 hours to 15. Both comparisons used samples of messages the tool itself had marked urgent, so they say nothing about urgent messages it missed. The study was the medical group's evaluation of its own tool. None of these studies is a randomized trial: they are a count of messages handled and before-and-after comparisons. Routing first is a reasoned bet, and it should be tested as one.
Defenders of AI drafting reply that time is the wrong measure. Doctors like the drafts: Stanford's clinicians reported less exhaustion, and the senior author of UC San Diego's 2024 study said that physicians "appreciated the help, which lowered cognitive burden." And routing carries a risk that drafting does not, since a person reads every draft before it is sent, while software that sends a message about chest pain to a scheduler acts where no doctor looks. The relief is real, but doctors reported it themselves, while the cost in time was measured, and it holds only while they check each draft as closely as a reply of their own. Nor is misjudged urgency a hazard of routing alone: most of the harmful drafts in the Mass General Brigham experiment misjudged or misstated how urgent the problem was.
Health systems, and the physicians on the committees that run their inboxes, should put routing first; Kaiser's two medical groups show it can run at scale. No router should be the only safeguard: where it hands messages to schedulers, a person should read each one, with a rule and a direct line for passing any symptom to a nurse at once. The router should be set, as Kaiser's in Southern California was, to flag too much rather than too little, and a clinician should check a sample of its choices each month and report to the medical staff how many urgent messages went to the wrong desk.
Drafting should come second, switched on for one kind of message at a time and kept only where an audit shows a gain. Replies about scans, test results and diagnoses, which took UC San Diego's doctors longest to fix, should be the last to get drafts. The order itself should be tested, by switching the features on clinic by clinic in random order and counting minutes, errors and missed urgent messages, with the results published.
At UC San Diego the commonest change doctors made to a draft concerned an appointment, and the costliest concerned what a scan showed. Software that sends requests for appointments to the people who book them could leave doctors more time to explain the scans, which is where a patient most needs a doctor's own words.
- Physician edits to AI-drafted replies at UC San Diego HealthPoursoltan and colleagues (NEJM AI, vol. 3, no. 9, Aug. 27, 2026) studied physician edits to AI-drafted replies to patient messages integrated in the Epic EHR at UC San Diego Health: 13,632 responses from 1,110 physicians to messages sent between April 2024 and August 2025, a retrospective cohort (2 Minute Medicine). A 15-category taxonomy of edits was built with a large language model and iterative expert review. Response time, from reply initiation to sending, was a workload proxy, examined with mixed-effects regression adjusting for message complexity and temporal factors; replies above the 95th percentile were excluded. All 15 categories were associated with longer response times. Healthcare Innovation calls the largest "marginal increases": interpreting radiology results (70.1%; 95% CI, 57.3 to 83.9%, per 2 Minute Medicine), clarifying or ruling out diagnosis (63.9%) and interpreting laboratory results (60.8%). The summaries consulted do not state the reference group. The most frequent edits involved scheduling or rescheduling appointments (38.5%). Single center.
- Earlier studies of drafted repliesUC San Diego (Tai-Seale and colleagues, JAMA Network Open, April 2024): a randomized waiting-list quality improvement study, June to August 2023, of 52 primary care physicians, with 70 contemporary controls. Drafts were associated with a 21.8% increase in read time (95% CI, 5.2% to 41.0%), a -5.9% change in reply time (95% CI, -16.6% to 6.2%; P = .33) and a 17.9% increase in reply length. Stanford (Garcia and colleagues, JAMA Network Open, March 2024): a five-week single-group quality improvement study, July 10 to Aug. 13, 2023, of 162 clinicians; mean draft utilization 20%; no change in reply action, write or read time; task load and work exhaustion scores fell among the 73 (45.1%) who completed both surveys. Children's Hospital of Philadelphia (Applied Clinical Informatics, April 2025; 323 participants, August 2023 to July 2024): clinicians saved 3 seconds per message, support staff 23. NYU Langone (npj Digital Medicine, October 2025; audit logs of 75 professionals): turnaround time fell 6.76%.
- Errors in drafted replies: two experimentsChen and colleagues at Mass General Brigham (The Lancet Digital Health, 2024) had GPT-4 generate 100 scenario and question pairs for patients with cancer; six radiation oncologists answered them and then edited GPT-4's drafts. The assessing physicians judged that the drafts posed a risk of severe harm in 11 (7.1%) of 156 survey responses and of death in one (0.6%); the hospital's release gives these as shares of LLM-generated responses. The majority of harmful responses incorrectly determined or conveyed the acuity of the scenario and the recommended action. Physicians reported that the drafts improved subjective efficiency in 120 (76.9%) of 156 cases. Dartmouth researchers (Association for Computational Linguistics meeting, July 7, 2026) built a tool that compares AI-generated replies with a dataset of real responses developed with Dartmouth Health professionals, and used it on drafts from Claude, Gemini, ChatGPT, Llama, Aloe and Qwen; separately they analyzed 146,000 conversations between 10,105 patients and their primary care physicians. They report that AI-generated answers frequently misalign with what clinicians would actually write. Editing time was not measured.
- Kaiser Permanente Northern California: messages sorted before the physicianLiu and colleagues (JAMA Network Open, 2024;7(4):e244867), a quality improvement study, describe the Desktop Medicine Program, in which natural language processing labels message content and sorts messages to teleservice representatives, medical assistants, pharmacists or physicians before they reach individual physician inboxes; Kaiser Permanente Northern California began the program in 2019, and the software was integrated into the electronic health record in May 2022. Of 4,835,200 messages evaluated by regional Desktop Medicine staff between April 1 and Aug. 31, 2023, 1,541,657 (31.9%) were resolved before reaching individual physician inboxes. Kaiser Permanente's Division of Research gives the resolvers as regional medical assistants or teleservice representatives (21.4% of total), physicians (7.2%) and pharmacists (3.3%). Of 4,709,261 messages in the period, 359,144 (7.6%) carried the label for emergent content for expedited review. The authors note that most messages were felt to be most appropriately addressed by the intended physician recipient.
- Kaiser Permanente Southern California: the Smart Messaging ToolNguyen and colleagues (JAMA Network Open, 2026;9(1):e2553174), a quality improvement study of the tool's first two years (March 28, 2023, to March 27, 2025): 3,030,247 messages from 1,042,418 patients, each labeled in the receiving clinician's inbox. Physicians judged 81.0% of 738 sampled labels correct (95% CI, 77.8% to 83.6%), against 44.0% of 500 under the legacy system, in which patients chose a subject from a preset menu ("other," 26.7%, counted as inaccurate). In departments that adopted the tool in early 2024, median time to first read of messages it labeled high acuity fell from 22.03 to 5.02 hours (samples of 4,827 messages from July to December of 2023, labeled retroactively, and of 2024); in comparable departments without it, from 18.38 to 14.58 hours (difference in differences, -11.01 hours; 95% CI, -11.95 to -9.84). Khang Nguyen, MD, the last author, told Becker's Health IT: "If we’re looking for financial ROI, I would actually say we haven’t found one."
- NYU Langone: an urgency flag for nurses' message poolsYang and colleagues (JAMIA Open, 2024;7(3):ooae078, Aug. 16, 2024), an observational quality improvement study, trained a BERT-based model on 40,132 patient messages to flag high-acuity topics that warrant an immediate callback, and deployed it in two shared message pools managed by registered nurses. On an expert-reviewed test set of 7,260 messages, the area under the ROC curve was 0.97 and average precision 0.72. After a brief pilot at 95% sensitivity, the team chose a threshold with 67% precision and 63% sensitivity, "with a compromise on missing some true urgent cases"; all messages are still actioned, and the flag changes only the review order. In a pre-post analysis of 396,466 messages (September 2021 to May 2022 against September 2022 to May 2023), a difference-in-differences analysis attributed reductions in median read time, counting only business hours, of 9 minutes for flagged messages sent during business hours and 21 minutes for those sent outside them. The authors judged randomization not feasible.Sources: PubMed, JAMIA Open (PMC)
- A company-funded routing studyAnderson and colleagues (NEJM AI, vol. 2, no. 3, Feb. 27, 2025): six of eight authors, including the first and last, list Switchboard, MD, as an affiliation; the other two are at Emory University and the Georgia Institute of Technology, and the first and last authors also list Emory. The natural language processing model was deployed prospectively in an outpatient clinic environment (four outpatient sites, according to 2 Minute Medicine, which calls the design a prospective cohort) and labeled and routed 469 conversations over 14 days into five classes (urgent, clinician, refill, schedule or form), compared with 402 unrouted conversations from the same period. Precision was at least 96.9%, recall at least 95.0% and total accuracy 97.8%. Staff took less time to first address a new message, and conversations were completed sooner (difference in medians, -22.5 hours; 95% CI, -36.3 to -17.7). Funded by Switchboard, MD.
- A general-purpose model triaging eye clinic messagesResearchers from Henry Ford Hospital, Virginia Commonwealth University School of Medicine and the Wilmer Eye Institute at Johns Hopkins (Journal of Clinical Medicine, 14(7):2395, March 31, 2025) de-identified 139 patient messages sent to a general eye clinic via MyChart and, in a retrospective study, compared triage by GPT-4 with triage by an ophthalmologist-in-training. Percent agreement was 64.7% for specialty clinic and 60.4% for triage acuity (Cohen's kappa 0.33 and 0.67). GPT-4 recommended a less urgent triage acuity in 6.5% of cases and an acuity equal to or sooner than the ophthalmologist-in-training's in 93.5%. "Of the nine cases where GPT-4 recommended a less urgent triage acuity, three could have resulted in patient harm." The authors conclude that such a system should complement rather than replace physician judgment.Source: Journal of Clinical Medicine