On a short leash

Software has begun to act for doctors, not just advise them. It should be kept on a short leash, because the doctor watching it rarely notices when it slips.

Illustration: a rubber stamp tied with a red string to a wall peg is hovering over paper forms on a wooden desk.

Over a single weekend, a piece of software working its way through CVS Health's computer systems cleared a backlog of documents that had built up for months. OpenAI described the feat on November 16, when it made its "computer-use" agents, programs that read a screen, click, type and move between applications much as a clerk would, available to clinicians and health systems. In its demonstration an agent read a patient's chart, drafted an MRI order, a clinic note and a reply to the patient, and filled in the insurer's prior-authorization form, the paperwork a plan demands before it will pay. Each piece then waited for a clinician to review it.

That waiting clinician is the safeguard on which this whole generation of software rests, and it is the weakest part of the design. Medical AI is moving from advising to acting, and the people deploying it assume that a busy clinician who glances at the result will catch what goes wrong. The evidence says otherwise. Agents should be given narrow powers in writing, a complete record of everything they do, and an auditor whose job is to watch the machine, not the patient.

The week made plain how quickly the acting has arrived. Besides CVS, Cigna Healthcare went from concept to deployment in under eight weeks, according to OpenAI. At the HLTH conference in Las Vegas, Suki, which made its name with ambient scribes, software that listens to a visit and drafts the note, launched agents that gather records for prior authorizations and hunt for gaps in documentation, and offered eligible customers a refund of up to the full fee if agreed targets are missed. Amazon put AI voice agents into general release that book and cancel appointments, take refill requests and reset patients' portal passwords around the clock; Temple Health in Philadelphia already uses them to verify callers on thousands of calls a week. OpenAI says reviewers rated 99% of 325 tasks done by its model as safe. That is the vendor's own number, from its own tests.

What happens to the human reviewer was shown by a study published on November 13 in npj Digital Medicine. Researchers in Marburg, Germany, gave 48 participants, from final-year medical students to experienced physicians, an AI assistant to help them classify heart sounds, then quietly made the assistant worse in stages until it was useless. Their accuracy fell from about four answers in five to fewer than three in five, below what they could manage unaided. Only one in eight noticed the decline without being asked, and making them commit to their own answer before seeing the machine's did not help. The authors concluded that "human oversight alone may be insufficient to reliably detect system malfunctions" when doctors listen to the heart with a machine's help.

The economics push in the same direction, because the point of buying an agent is to need fewer people. Roberta Schwartz, Houston Methodist's chief innovation officer, told Becker's that she asks vendors directly what reduction in staff their technology has produced, and that time savings alone do not meet her bar. "We aren't going to be able to pay for the innovations and keep all the people," she said. A Vizient survey of 416 hospital leaders found that 17% of the organizations reporting the best returns from AI had already made AI-related layoffs, though the returns were the hospitals' own unaudited estimates. The checking falls to whoever is left.

Washington is keen to hand over more. A roadmap posted by Medicare's administrators says that many parts of the annual wellness visit "could potentially be delivered via clinical SI tools", SI being the document's term for AI, and that it anticipates letting AI-led services take part in its payment experiments. It promises "appropriate human oversight" and "continuous real-world monitoring" without spelling out what that oversight should be. Meanwhile the FBI warned on November 13 that attackers are using an open-source platform of AI agents against health care, and the American Hospital Association's cybersecurity adviser, John Riggi, said some intrusions had gone from first probe to privileged access in under three hours. Software that holds a clinician's log-ins to the record and the insurer's portal is a prize for attackers moving at that pace.

Vendors will reply that the controls already exist. OpenAI offers role-based permissions and audit logs; Ambience, one of its partners, demonstrated an agent that stops and asks a clinician when the record is ambiguous, with hospitals choosing which actions it may take and able to pause it at any moment. Those controls are real, but every one of them is optional. A permission is only as narrow as the hospital that sets it, a log protects nobody if no one reads it, and the Marburg study suggests the approval click is the step most likely to fail precisely when the software starts to fail. The hospitals buying agents to cut staff are unlikely to spend the savings on more checking unless someone requires it.

Hospitals and medical staffs should therefore write down, before any agent is switched on, which actions it may take, and require a named person to approve anything that leaves the building: an order, a prior-authorization request, a message to a patient. They should have someone outside the clinic, in pharmacy or quality, audit a sample of each agent's logged work every week, and test it occasionally with planted errors, as the German researchers did. Physicians should ask whose log-in an agent is using and decline to let it act under their names without that record. Vendors should publish error rates from real deployments rather than demonstrations. Medicare should define "appropriate human oversight" before it pays for care led by AI, not after.

A machine that can clear months of paperwork in a weekend can make months of mistakes in one. Someone should be paid to notice.

Comments on the FDA's draft guidance on robotically assisted surgical devices are due Nov. 24, and RSNA meets in Chicago from Nov. 29 to Dec. 3. Comments on Sen. Bill Cassidy's PATIENT AI Act discussion draft, which would create a Medicare payment pathway for AI tools, are due Dec. 1, and the FDA holds a public workshop on autonomous and remotely operated robotic devices on Dec. 2 and 3. The Federation of State Medical Boards plans to release draft guidance on accountability when AI performs clinical work for public comment in December.

The Medicare Payment Advisory Commission meets Dec. 10 and 11. Comments on CMS's proposed anti-fraud rule, which would require clinicians who bill Medicare Advantage plans to enroll in traditional Medicare, are due 60 days after its publication in the Federal Register. On Jan. 1, Medicare Advantage, Medicaid and some exchange plans must have electronic prior-authorization interfaces running, though a survey found only 1 in 10 payers fully ready.

Latest news

  1. Medicare Advantage plans would have to make in-network clinicians file prior authorizations through an electronic interface, CMS proposes
  2. Medicare's first mandatory Advantage model would swap star ratings for local 'Diamond Ratings' in randomly chosen states from 2029
  3. A California health system says an employee uploaded emergency patients' diagnoses and record numbers to ChatGPT

See all the latest news