Out of the loop

"Do not mention Lam reduction plan." That, according to a lawsuit filed in San Francisco on October 27, is what ChatGPT told Bowen Wan, a 31-year-old software engineer with epilepsy, before an appointment with her primary-care doctor. Her parents allege that the chatbot had drawn up a 25-day schedule to take her off lamotrigine, the seizure drug she had taken for years, and that she died on or about the day it had her fully off. The claims are unproven; OpenAI called the case "incredibly heartbreaking," TechCrunch reported. In the complaint's telling, the one person placed to catch the error was told nothing.

Medicine answers every worry about AI the same way: a clinician reviews every denial, a physician signs every note, a radiologist confirms every flag. The week's evidence suggests that the human check is mostly assumed rather than measured, and that where it has been measured it catches less than its reputation implies. Anyone who deploys AI in medicine on the promise of human review should have to publish how often that review catches the machine's mistakes.

A review in npj Digital Medicine pooled 46 reports on clinicians given automated advice and found that their odds of reaching the right decision were about ten times higher when the advice was right than when it was wrong. In real-world studies of computer-read electrocardiograms, about a quarter of the machine's errors went neither corrected nor noticed. The authors concluded that human oversight "should be evaluated as an error-interception capability rather than assumed to ensure safety." In an experiment in The Lancet Digital Health, 125 doctors and trainees, 50 of them radiologists, read head CT scans for bleeding with AI advice that was deliberately wrong a fifth of the time. They followed bad advice and also dismissed good advice, and the dismissing cost more accuracy.

Medicare's experiment with AI in prior authorization, called WISeR, rests on the same promise: a clinician reviews every denial. A research letter in JAMA found that in the six pilot states, use of epidural steroid injections fell by about 15% relative to the rest of the country in the first six months, and use of the other targeted services by about a fifth. Its authors say the data cannot show whether the forgone care was needed. Records obtained by the Electronic Frontier Foundation show that the Centers for Medicare & Medicaid Services sent requests for information or corrective action plans to four of the pilot's six contractors, and that the agency itself described the contractors as having "an incentive to deny as many claims as possible." What nobody has published is how often the reviewing clinicians overrule the software.

The doctor who reviews an AI-drafted note is another unmeasured checker. At Beth Israel Lahey Health, a study of 902 outpatient clinicians in Annals of Internal Medicine found that enrolling in an ambient scribe cut about a minute of note time per visit, with no change in after-hours charting or in measured productivity, while heavy users' notes grew longer. The authors could not tell which text the machine had written. In 20 emergency departments, physicians using a scribe spent 2.4 fewer minutes per note on notes about 14% longer. Doctors are spending less time on more text: either the drafts are very good or the reading is getting thinner, and no one yet knows which.

Where institutions do measure, they act. Inova Health System in Virginia reviews its AI features every quarter and drew up plans to switch off three whose performance had declined. Washington is moving the other way. The final version of a federal health IT rule, known as HTI-5, strikes the requirement that certified record systems let users see how their predictive tools were trained and tested, information that any checker needs; regulators cited its "limited demonstrated benefit." And at Johns Hopkins Bayview Medical Center, a worker uploaded a spreadsheet of 3,182 patients' details to an AI tool the hospital had not approved.

In a Pew Research Center survey published on October 29, three in five Americans said they would be uncomfortable with AI in their own care, and on diagnosis nearly half wanted a human doing the final review while hardly anyone wanted the machine to decide alone. The reviewer is what the public has agreed to buy.

Skeptics will say that human review was never meant to be perfect, and that the fair comparison is a doctor with AI against a doctor without it. On that comparison the pair sometimes wins: in the head CT experiment, correct advice with markings on the scan improved doctors' accuracy. Auditing the reviewers, they add, piles paperwork on clinicians already buried in it. These points have force, but they argue for measurement rather than against it. The only way to show that a doctor with AI beats a doctor alone is to measure the pair in practice, and the basic figures, how often reviewers overrule the machine and how often appeals reverse it, already sit in the systems that run the tools. A contractor paid a share of the value of the requests it denies is the last party that should grade its own reviewers.

Medical staffs should require, as a condition of approving any AI tool that a clinician must check, a regular audit of how often the checking catches known errors, and should retire tools that fail, as Inova did. Hospitals should give staff approved tools for everyday tasks, so that patient spreadsheets stay out of unvetted ones. CMS should publish WISeR's denial rates, the rate at which its clinicians overrule the software and the rate at which appeals succeed, contractor by contractor. HHS should restore the requirement that record systems disclose how predictive tools were tested. States that require a physician to review AI denials should also require the overturn rates to be reported. And physicians should ask patients, as routinely as they ask about supplements, whether a chatbot has advised them to change a medicine.

The chatbot in the Wan complaint, if the family's account is right, worked to keep a doctor out of the loop. The rest of medicine's AI keeps the doctor in it, on paper. Whether the doctor is doing anything there is a question that can be answered with data, and it should be answered before a court asks it.