Physician in the Loop
For youSign in
The Friday letter ยท

Show your work

Software is gaining the power to prescribe, triage and code faster than anyone is checking its results. It should earn that power by publishing its errors.

A dark navy equipment case with a carrying handle has a glass window in its front, through which a glowing amber gauge and two gray gears are visible.

An adult in Utah with mild acne can now answer a questionnaire, scan their face and receive a prescription chosen by software, for $4.99 a month. Under the agreement the state signed with Nolla Health, two physicians check every prescription before it is sent, at first. Once the software meets its targets, review thins until a physician samples about one case in ten each month. In the same week Utah approved a second company's software to recommend renewals of diabetes and heart drugs, with a physician signing each one.

What nobody outside these companies can yet say is how often the software is wrong. Nolla's figure, that clinicians agreed with its choices in more than 96 percent of real-world cases, is its own. The week's news showed the same gap across medicine: tools are being given authority over patients faster than anyone measures what they do with it. Software should earn that authority only by publishing its error rates, checked by someone other than its maker, for as long as it stays in use.

Approval is no proxy for evidence. Researchers in Helsinki looked at 77 AI products authorized in Europe or America to read pathology slides and blood smears, and found published performance data for fewer than two in five. The authors noted that such results do not have to be published at all. Two of Medicare's regional contractors, covering 20 states, drew the obvious conclusion about another class of cleared products. They proposed to stop paying for AI that measures brain structures on MRI, on the ground that "there is insufficient evidence of clinical utility or validity." A clearance says a tool measures what it claims; it does not say patients are better off.

Where hospitals do check, they find things. Five health systems told Becker's Hospital Review they were holding off on some of Epic's AI tools, and Children's Healthcare of Atlanta paused one after it produced "too many inaccuracies across diagnosis and patient locations." That was caught because one hospital looked. Meanwhile the oldest check in the record itself failed. Flaws in Epic's software, found by testing with an AI model, let some configurations open a patient's chart without leaving an entry in the audit trail, the log that shows who looked. Epic's security chief told hospitals that when they think they are patching fast enough, they "need to patch faster."

Patients are not waiting for the evidence either. At Advocate Health, the number of patients linking their records to ChatGPT rose more than 7,000 percent between July and August. The health system did not arrange it, cannot see what individual patients do with it and bears costs that rise with every request. Its chief information officer described "a disconnect between who benefits and who pays." Novant Health, meanwhile, is selling 36 urgent care centers as it moves patients toward round-the-clock virtual care in which an AI assistant gathers symptoms before a clinician joins the visit.

Britain has drawn the right lesson. Its government accepted all 44 recommendations of a national commission on regulating AI in health care, promising "less reliance on point-in-time pre-market assessments," a public database of adverse incidents and a review of how responsibility is divided between manufacturers and the providers who use their products. The principle is sound: judge a tool by what it does to patients after it is approved, and let doctors see the record.

The strongest objection is that this asks more of software than of doctors. No physician publishes a personal error rate, the tools spread because doctors are scarce and tired, and a reporting burden would slow useful products while favoring the vendors big enough to bear it. But physicians are checked continuously: licensed, credentialed, reappointed after peer review and sued when they err. Software doing a doctor's work faces far less of that once it ships. And the burden is smaller than it sounds. Utah's agreement already requires monthly reports, adverse events within a day and quarterly reports the state makes public, and six outside evaluators, among them the Coalition for Health AI and Stanford, will inspect decision logs. If a startup selling acne treatment by app can manage that, so can a vendor selling to the largest health systems in the country.

Medical staffs should credential AI tools the way they credential physicians: no tool that drafts notes, orders or triage decisions without the vendor's error rates on local patients, rechecked at every reappointment. Hospitals should test before rollout, as Atlanta did, and publish what they find. Utah should publish its evaluators' findings in full beside each company's claims. Medicare's contractors should hold their line on the brain MRI tools until the evidence arrives, and Medicare should ask for the same before it pays for the AI services the CPT panel has just given codes. The FDA, whose comment period on generative AI devices closes Oct. 19, should copy Britain's public incident database. Physicians asked to sign behind a machine should ask for its error rate first.

Nolla will have to tell Utah, every month, how often its physicians overruled the software. Every tool that writes in a patient's chart should have to say the same.

SOURCE MATERIAL
LOOKING AHEAD

Medicare open enrollment opens Oct. 15 and runs through Dec. 7. The American Society of Anesthesiologists meets in San Diego for ANESTHESIOLOGY 2026 from Oct. 16 to 20, and comments on the FDA's discussion paper on generative-AI medical devices are due Oct. 19. The HIMSS AI in Healthcare Forum meets in San Diego on Oct. 22 and 23, with a session titled "Human in the Loop: Myth or the Essential Path Forward?" according to Healthcare Finance News.

Noridian holds an open meeting on its proposal to stop covering AI brain MRI tools on Oct. 22 and Palmetto GBA on Oct. 26; written comments on both close Nov. 14. Comments also close Oct. 26 on Colorado's rules for automated decisions and chatbot safety. The final 2027 Medicare physician fee schedule is expected around Nov. 1, the same day Prisma Health's purchase of Novant's urgent care centers is expected to close.