Aim for better, not just sooner
AI health products tend to report how many people they reached and how quickly. The test that matters is whether those patients did better, and buyers should ask before they scale.
Listen to this special topic
read by an AI voiceIn Dresden, neurosurgery patients who got a checked, plain-language AI summary of their findings before their first visit had shorter visits and rated them higher. In a randomized trial of 300 adults published on Oct. 15, the median consultation ran 25 minutes with the summary and 33 without. The last author co-founded the company whose software wrote the summaries, though the university says he took no part in collecting or analyzing the data. Their abstract ends on the sentence that matters. Effects on clinical outcomes and on doctors' workload, they write, "require confirmation before routine adoption."
That sentence is the right test for AI in medicine. Shorter visits, shorter waits, more patients seen and more questions answered all have value. Success is patients who do better, with fewer deaths, complications, admissions and missed diagnoses. Insurers and health systems should ask for that evidence before they spread a tool, not after.
Trials of AI mostly ask something else. A 2024 review of 86 randomized trials of clinical AI found that four in five reported a positive main result, mostly on how much the software found or how well it performed. Of 4,667 studies of medical AI gathered from a year of systematic reviews, about one in forty was randomized. Utah's office of AI policy says its pilot with Doctronic, a company whose software renews prescriptions, could "expand patient access." It began by collecting safety data, with economic and public health outcomes planned.
A randomized trial in Kenya shows the gap. This summer Vyonne Njeri, a clinical officer in Nairobi, told NPR about an infant she had taken for a cold until the software reading her notes prompted her to check his heart. "That's something I would have missed on any other day," she said. Her story came after the trial. Across the trial's 16 clinics in 2025, officers with the software wrote better notes and recorded an appropriate diagnosis more often. By the main test, their patients did no better. Treatment failure, anything from a return visit with symptoms unresolved to a missed diagnosis or death, struck about two patients in a hundred in each group within two weeks. The assistant was safe, the authors concluded, but "did not reduce treatment failure within 14 days and any benefit, if present, is probably modest."
An earlier study at Penda Health, the chain that runs those clinics, found the same split. It was a quality improvement study, paid for by OpenAI, which took part in the analysis. Clinicians with the tool made 16% fewer diagnostic errors and 13% fewer treatment errors, as judged by doctors reviewing the records blind. "There were no statistical differences in patient-reported outcomes," the authors wrote. The Kenyan trial's authors suggest such gains may reach patients eventually, if the theories behind them hold. Neither study could show that they had.
Sometimes the answer is yes. In a randomized trial of nearly 16,000 hospital patients by researchers in Taipei, software read each patient's electrocardiogram and warned doctors about those at high risk of dying. Within three months, about seven fewer patients in every thousand had died in the group whose doctors got the warnings. In a randomized trial across 74 hospital units at two health systems, software that read nurses' charting for signs of decline lowered the moment-to-moment risk of dying in the hospital by about a third, and more patients were moved unexpectedly to intensive care, the kind of early escalation its authors say can improve survival.
Speed has a serious defense. By OpenAI's own analysis, ChatGPT averaged more than 580,000 health messages a week in late 2025 from places over half an hour's drive from a general hospital. For someone who cannot get an appointment at all, an answer tonight beats a perfect trial in five years, and demanding proof can keep people waiting. The wait does real harm. But an answer is worth what the patient does with it. In a randomized study run in 2024, in which 1,298 adults in Britain worked through medical scenarios written by doctors, those using a chatbot named a relevant condition in about a third of cases at best, less often than people using any source they liked. Tested alone, the chatbots almost always named one. Even OpenAI's report says AI "will not, on its own, reopen a shuttered hospital."
So ask where asking is cheapest, before a pilot becomes a contract. Proof need not be slow, though it may need to be large. The Kenyan trial enrolled 9,691 patients in under three months, and its authors report that showing a modest effect on rare events such as hospitalization and death would take more than 100,000. A health system can switch a tool on for some clinicians and not others, as that trial did, and follow one patient result beside the access figure, such as admissions or missed diagnoses. Insurers covering more than 165 million Americans have pledged to offer outcomes-based payment for technology-supported care by Jan. 1, 2028, in line with a Medicare model that bases payment on the share of patients meeting their outcome targets. They should hold every AI tool and AI-run service to the same rule, and health systems every tool they buy. Utah should have the outcomes it plans to add before any group of Doctronic's drugs reaches the phase in which the software sends renewals straight to the pharmacist.
The Dresden visits were eight minutes shorter, and patients felt better informed. Whether any of them recovered better is a question the trial was not built to answer, and its authors said so. Faster care is a means to that end. Until a tool shows that its patients did better, its speed is only a promise, and nobody should buy it at scale.
- The Dresden trial of AI summaries before a visitBenker-Hasani, Richter and colleagues, all of the Department of Neurosurgery at University Hospital Carl Gustav Carus, TUD Dresden University of Technology, reported the AI-INFOCARE randomized controlled trial in NEJM AI on Oct. 15, 2026. In a single-center trial in an outpatient neurosurgical setting, 300 adults were randomly assigned 1:1 to standard care with or without a quality-checked, AI-generated layperson summary of their medical findings (Simply Onno) before consultation. The primary end point, patient-reported quality of physician-patient interaction, scored 4.42 against 4.25 (difference 0.17; 95% CI 0.05 to 0.29; P=0.005). Median consultation time, a secondary end point, was 25 against 33 minutes; secondary confidence intervals were unadjusted for multiplicity. The university's release of Oct. 23, 2026 says 17 physicians took part, that the researchers did not examine long-term clinical implications, and that the last author, Witold Polanski, is a co-founder and medical director of Simply Onno who was not involved in patient interviews, data collection or statistical analysis.
- The Kenyan trial of an AI assistant for clinical officersAgweyu, Mwaniki and colleagues (Nature Medicine, published June 26, 2026) ran a pragmatic, cluster-randomized trial in 16 primary care facilities in Kenya, randomizing clinical officers to the electronic medical record with or without assistance from GPT-4o (May 2025 release). From April 22 to July 16, 2025, 9,691 patients were enrolled under 103 clinical officers. Treatment failure within 14 days, expert-adjudicated, occurred in 102 of 4,693 patients (2.2%) with assistance and 94 of 4,654 (2.0%) without (adjusted odds ratio 0.77; 95% CI 0.55 to 1.08; P = 0.13). Assisted officers were more likely to record an appropriate diagnosis (adjusted odds ratio 1.74). No serious adverse events were judged related to the intervention. The Gates Foundation funded the trial and PATH sponsored it; OpenAI gave in-kind support and, the paper says, had no role in design, data collection or analysis. NPR reported the adjusted odds ratio as a 23% decrease in treatment failures that was not statistically significant; the unadjusted rates above were slightly higher with assistance.
- NPR's report from NairobiJoseph Kim reported for NPR on July 23, 2026, under the headline "This AI tool promises a 'second pair of eyes' to clinicians. Did patients benefit?" The tool, called AI Consult, checks clinical officers' electronic notes in the background and shows green, yellow or red prompts; it was tested across 16 Kenyan primary clinics operated by Penda Health. Vyonne Njeri is described as a registered clinical officer in Nairobi who sees patients on her own like a nurse practitioner; on a visit a few months before the story, the tool prompted her to check the heart of a 4-month-old boy with an elevated heart rate whom she had taken for a cold. Dr. Bilal Mateen, a co-author and chief AI officer at PATH, said treatment failures "are just too rare in primary care," and that a trial would need about 139,000 people to detect a meaningful difference. NPR notes that the Gates Foundation, which funded the trial, provides financial support to its global health team.Source: NPR, July 23, 2026
- The earlier study at Penda Health, funded by OpenAIKorom and colleagues, of Penda Health, Nairobi County and OpenAI, posted "AI-based Clinical Decision Support for Primary Care: A Real-World Study" as a preprint on arXiv (2507.16947). They describe a quality improvement study of 39,849 patient visits at all 15 Penda clinics in Nairobi County, in which half of the clinicians in each clinic were randomly assigned access to AI Consult; the primary evaluation period ran from March 1 to April 18, 2025. A panel of 108 physicians, blinded to group, reviewed a random sample of 5,666 visits. The AI group made 16% fewer diagnostic errors and 13% fewer treatment errors. The share of patients who still felt unwell seven days after the visit was 4.3% without AI and 3.8% with it (p = 0.234), which the authors describe as no statistical difference; they write that the study was not powered to detect an effect of that size. Funding was provided by OpenAI, which was involved in the analysis and reporting.Source: arXiv preprint 2507.16947
- Randomized trials that followed deathsLin and colleagues of Tri-Service General Hospital and the National Defense Medical Center in Taipei (Nature Medicine, April 29, 2024) tested an AI-ECG alert, a report and warning messages to physicians flagging patients predicted to be at high risk of death, in a multisite trial with 39 physicians and 15,965 patients: 3.6% of the intervention group and 4.3% of controls died within 90 days (hazard ratio 0.83; 95% CI 0.70 to 0.99). Rossetti and colleagues (Nature Medicine, April 2, 2025) cluster-randomized 74 clinical units at two health systems to the CONCERN early warning system, built on nursing documentation patterns, or usual care; across 60,893 encounters, the intervention group had a 35.6% decreased instantaneous risk of death (adjusted hazard ratio 0.64), 11.2% decreased length of stay, a 7.5% decreased instantaneous risk of sepsis and a 24.9% increased instantaneous risk of unanticipated intensive care unit transfer. The authors write that early recognition and escalation of treatment, including transfer to the ICU, can improve outcomes.
- What AI trials have testedHan, Acosta, Shakeri, Ioannidis, Topol and Rajpurkar (The Lancet Digital Health, May 2024) reviewed 86 randomized controlled trials of AI in clinical practice: 70 (81%) reported positive primary endpoints, primarily related to diagnostic yield or performance, and the authors raised concerns about the predominance of single-centre trials and recommended that future trials prioritise patient-relevant outcomes. Wang and colleagues (npj Digital Medicine, April 30, 2026) drew 4,667 primary studies from 218 systematic reviews published between September 2023 and September 2024: 88.2% were preclinical and 113 (2.4%) were randomized controlled trials, of which 67.3% were single-center and 82.3% reported favorable outcomes.
- Chatbots, the public and the access caseBean and colleagues, led from the Oxford Internet Institute (Nature Medicine, Feb. 9, 2026), recruited 1,298 adults living in the UK, gave each up to two of ten text scenarios developed with three doctors, and randomly assigned them help from a language model (GPT-4o, Llama 3 or Command R+) or a source of their choice; data were collected from Aug. 21 to Oct. 14, 2024. Tested alone, the models identified conditions in 94.9% of cases; participants using them identified relevant conditions in fewer than 34.5% of cases and were significantly less likely than the control group to name at least one (the control group had 1.76 times the odds), while their choice of what to do next was no better than the control group's. OpenAI's January 2026 report, AI as a Healthcare Ally, says that over a four-week span in late 2025 ChatGPT averaged more than 580,000 healthcare-related messages a week from hospital deserts, which it defines as locations more than a 30-minute drive from a general medical or general children's hospital.
- Utah's renewal pilot and Medicare's ACCESS modelUtah's Office of Artificial Intelligence Policy says that in the first phase of its agreement with Doctronic a licensed physician must review each AI-recommended renewal before it is sent to a pharmacist, and in the second the AI tool may submit renewals directly to the pharmacist, who may escalate to a physician; each medication group becomes eligible for the second phase once Doctronic has filled 250 medications in that group. Data collection initially prioritized core safety metrics, with planned expansion to economic and public health outcomes. The Centers for Medicare and Medicaid Services says its ACCESS model began July 5, 2026, runs 10 years through June 30, 2036, covers conditions including high blood pressure, diabetes, chronic musculoskeletal pain and depression, and will base payment on the share of an organization's patients who meet their outcome targets. CMS says payers representing more than 165 million Americans have pledged to offer outcomes-based payment arrangements aligned to the model by January 1, 2028.