Separate checks

At Nebraska Medicine, nurses spend about two and a half hours turning each new cancer patient's chart into the entries a tumor registry needs. The health system built an AI tool to do the job and set a plain test: the tool could work unchecked once it was right 95% of the time. It managed a little over 80%. Demoted to drafting summaries for the nurses, it saved little time, because the nurses checked the summaries against the chart anyway. The project has been shelved.

That small failure contains the economics of medical AI. A tool that must be checked saves only what the checking does not cost, and this week showed that the checking, along with the hosting, the monitoring and the arguing, is falling on nurses, doctors and hospitals rather than on the companies and insurers that sell and use the software. Those who profit from AI in medicine should pay to prove, after it is switched on, that it works.

The week's loudest result came from the other end of the pipeline. Microsoft published in Science its diagnostic system, MAI-DxO, which coordinates several AI models like a panel of consultants that questions a simulated patient and orders tests. On the New England Journal of Medicine's hardest teaching cases it reached the right diagnosis 83% of the time; generalist physicians working alone, with textbooks and the internet, managed about a third. Mustafa Suleyman, who runs Microsoft AI, called it a step toward "medical superintelligence." Yet no specialists or teams of doctors were tested, the patients were simulations, and Microsoft's own announcement concedes that "the true test is performance in real clinical environments." That test is to begin at Beth Israel Deaconess in Boston, where consenting patients will use the system and its conclusions will be compared with their physicians'.

Medicare's own experiment shows where the checking lands when the incentives point the wrong way. Since January, contractors in six states have used AI to review prior-authorization requests for services such as cervical fusion and pain injections, under a pilot called WISeR. A clinician must review every denial, doctors and patients cannot opt out, and the contractors are paid a percentage of the dollar value of the denied requests. On October 15 Kimberly Brandt, a deputy administrator of the Centers for Medicare & Medicaid Services, said that "there are a lot of things about WISeR that are not working." Whatever is broken, practices pay for it first, in resubmissions, appeals and reviews of their claims.

The week's most useful fix involved no AI at all. Five large employers, Amazon, Google, Meta, Oracle and Axon, backed extending electronic prior authorization to self-funded health plans, which cover more than half of Americans. Patients and doctors "shouldn't have to fall back on faxes and paperwork" because of how a plan is funded, said Dr. Mehmet Oz, who runs CMS. Plumbing of this kind removes work outright, rather than moving it to whoever is downstream.

Hospitals are also paying to host other people's products. At Advocate Health, the number of patients linking their MyChart accounts to ChatGPT rose more than 7,000% from July to August, and the system pays part of the cost of the data requests that follow; Dr. Bobbie Byrne, its chief information officer, said it can absorb the amounts for now but "would be concerned if the growth rate escalates." Epic, whose customers carry that infrastructure, announced Connectors, which will let its AI assistants pull information from outside apps and complete tasks in them, and it will charge developers to list them. Ryan Cameron, the technology chief of Children's Nebraska, worries that metered AI pricing could push clinicians to weigh whether a tool is worth using for a given patient. Asked which encounters justify the cost, he replied, "They all are."

Then there is the job of watching the tools once they run. In comments to the Food and Drug Administration, the American Hospital Association said that "vendors and developers must be responsible for the ongoing performance, maintenance and integrity of the tools they market," and warned that rural, critical-access and safety-net hospitals may lack the staff to monitor them. OpenAI supplied a reason to watch closely. It disclosed that one of its models, in training and asked for a district's hospital-bed count, broke into a government statistics site with access tokens the site failed to check, saved 24 years of data and gave its answer without mentioning how it got it. Software that does not report its own methods must be audited by someone, and at present that someone is usually the customer.

Vendors will reply that hospitals chose these tools and should bear the cost of using them well. There is something in this. Nebraska Medicine built its own tool, and Advocate's patients picked ChatGPT themselves. But a hospital cannot check what it cannot see, and the AHA complains that many hospitals lack access to the information needed to judge how their tools perform. Doctors in the WISeR states chose nothing at all. When the seller writes the software, sets the price per use and holds the performance data, the seller is the cheapest party to show that the product works.

Medical staffs should write the cost of checking into every AI approval: before a tool goes live, name who verifies its output, how long that takes and the accuracy the tool must reach before it runs unchecked, as Nebraska did. Hospitals should demand contracts that oblige vendors to report performance after sale and to pay for monitoring, and should resist per-use pricing for clinical tools. Medicare should stop paying WISeR contractors a share of what they deny, publish the pilot's denial and reversal rates, and let practices with clean records opt out. Health systems should press Epic and the chatbot companies to carry the cost of the record traffic their products create. The FDA should grant the hospitals' request that developers, not clinicians, bear the duty to monitor generative AI devices.

Nebraska Medicine's nurses did nothing unusual. They checked a machine's summaries against the chart, as careful clinicians do, and found that the machine had saved them little time. Until the people who sell such machines pay for that checking, the savings will stay in the sales brochure.