On a short leash
Software has begun to act for doctors, not just advise them. It should be kept on a short leash, because the doctor watching it rarely notices when it slips.

Listen to this letter
read by an AI voiceOver a single weekend, a piece of software working its way through CVS Health's computer systems cleared a backlog of documents that had built up for months. OpenAI described the feat on November 16, when it made its "computer-use" agents, programs that read a screen, click, type and move between applications much as a clerk would, available to clinicians and health systems. In its demonstration an agent read a patient's chart, drafted an MRI order, a clinic note and a reply to the patient, and filled in the insurer's prior-authorization form, the paperwork a plan demands before it will pay. Each piece then waited for a clinician to review it.
That waiting clinician is the safeguard on which this whole generation of software rests, and it is the weakest part of the design. Medical AI is moving from advising to acting, and the people deploying it assume that a busy clinician who glances at the result will catch what goes wrong. The evidence says otherwise. Agents should be given narrow powers in writing, a complete record of everything they do, and an auditor whose job is to watch the machine, not the patient.
The week made plain how quickly the acting has arrived. Besides CVS, Cigna Healthcare went from concept to deployment in under eight weeks, according to OpenAI. At the HLTH conference in Las Vegas, Suki, which made its name with ambient scribes, software that listens to a visit and drafts the note, launched agents that gather records for prior authorizations and hunt for gaps in documentation, and offered eligible customers a refund of up to the full fee if agreed targets are missed. Amazon put AI voice agents into general release that book and cancel appointments, take refill requests and reset patients' portal passwords around the clock; Temple Health in Philadelphia already uses them to verify callers on thousands of calls a week. OpenAI says reviewers rated 99% of 325 tasks done by its model as safe. That is the vendor's own number, from its own tests.
What happens to the human reviewer was shown by a study published on November 13 in npj Digital Medicine. Researchers in Marburg, Germany, gave 48 participants, from final-year medical students to experienced physicians, an AI assistant to help them classify heart sounds, then quietly made the assistant worse in stages until it was useless. Their accuracy fell from about four answers in five to fewer than three in five, below what they could manage unaided. Only one in eight noticed the decline without being asked, and making them commit to their own answer before seeing the machine's did not help. The authors concluded that "human oversight alone may be insufficient to reliably detect system malfunctions" when doctors listen to the heart with a machine's help.
The economics push in the same direction, because the point of buying an agent is to need fewer people. Roberta Schwartz, Houston Methodist's chief innovation officer, told Becker's that she asks vendors directly what reduction in staff their technology has produced, and that time savings alone do not meet her bar. "We aren't going to be able to pay for the innovations and keep all the people," she said. A Vizient survey of 416 hospital leaders found that 17% of the organizations reporting the best returns from AI had already made AI-related layoffs, though the returns were the hospitals' own unaudited estimates. The checking falls to whoever is left.
Washington is keen to hand over more. A roadmap posted by Medicare's administrators says that many parts of the annual wellness visit "could potentially be delivered via clinical SI tools", SI being the document's term for AI, and that it anticipates letting AI-led services take part in its payment experiments. It promises "appropriate human oversight" and "continuous real-world monitoring" without spelling out what that oversight should be. Meanwhile the FBI warned on November 13 that attackers are using an open-source platform of AI agents against health care, and the American Hospital Association's cybersecurity adviser, John Riggi, said some intrusions had gone from first probe to privileged access in under three hours. Software that holds a clinician's log-ins to the record and the insurer's portal is a prize for attackers moving at that pace.
Vendors will reply that the controls already exist. OpenAI offers role-based permissions and audit logs; Ambience, one of its partners, demonstrated an agent that stops and asks a clinician when the record is ambiguous, with hospitals choosing which actions it may take and able to pause it at any moment. Those controls are real, but every one of them is optional. A permission is only as narrow as the hospital that sets it, a log protects nobody if no one reads it, and the Marburg study suggests the approval click is the step most likely to fail precisely when the software starts to fail. The hospitals buying agents to cut staff are unlikely to spend the savings on more checking unless someone requires it.
Hospitals and medical staffs should therefore write down, before any agent is switched on, which actions it may take, and require a named person to approve anything that leaves the building: an order, a prior-authorization request, a message to a patient. They should have someone outside the clinic, in pharmacy or quality, audit a sample of each agent's logged work every week, and test it occasionally with planted errors, as the German researchers did. Physicians should ask whose log-in an agent is using and decline to let it act under their names without that record. Vendors should publish error rates from real deployments rather than demonstrations. Medicare should define "appropriate human oversight" before it pays for care led by AI, not after.
A machine that can clear months of paperwork in a weekend can make months of mistakes in one. Someone should be paid to notice.
- On Nov. 16 OpenAI made computer-use agents, which click, type and move between applications, available through its API, ChatGPT for Healthcare and ChatGPT for Clinicians. In a demonstration an agent drafted an MRI order, an encounter note and a patient reply for clinician review and prepared a prior authorization request; OpenAI said 99% of 325 reviewed EHR inbox tasks were rated safe and physicians rated the agent very or extremely helpful in 93% of 646 assessments, in its own evaluation. CVS Health said one agent cleared a months-long document backlog over a weekend, and Cigna Healthcare moved from concept to deployment in under eight weeks.Sources: OpenAI, Becker's Hospital Review
- In a study published Nov. 13 in npj Digital Medicine, 48 participants (18 final-year medical students, 19 residents and 11 experienced physicians) classified 40 heart sounds with an AI assistant whose reliability was lowered in blocks from 90% to 60%, 50% and 0%. AI-assisted accuracy fell from 81.45% to 58.33%, below unassisted performance; only 12.5% reported noticing the decline without prompting, and randomly assigning participants to classify before seeing the AI's answer did not improve performance. The authors wrote that "human oversight alone may be insufficient to reliably detect system malfunctions" in AI-assisted cardiac auscultation.
- At HLTH in Las Vegas on Nov. 16, Suki introduced Suki OS, a platform connecting health care software systems and AI agents, with initial agents that identify prior authorization requirements and gather supporting documentation, and that find documentation gaps for coding and revenue cycle work. The company said eligible customers can get up to a 100% refund if agreed performance targets are not met, and that its technology is used by more than 450 health care organizations.Sources: Healthcare Dive, Fierce Healthcare
- Amazon Web Services said Nov. 17 that AI voice agents in Amazon Connect Health are generally available to book, reschedule and cancel appointments with real-time insurance checks, take prescription refill and renewal requests and reset patient portal passwords at any hour. Temple Health in Philadelphia, with more than 1.5 million outpatient visits a year, has handled "thousands of calls each week" with patient verification since June. AWS said routine and administrative requests make up around 60% of inbound calls, citing customer call data.Sources: Amazon Web Services, Becker's Hospital Review
- Roberta Schwartz, chief innovation officer of the eight-hospital Houston Methodist system, told Becker's Hospital Review in an interview published Nov. 12 that the system requires a 3-to-1 or 5-to-1 return on investment before adopting an innovation and asks vendors directly what headcount reduction their technology has produced. Time savings alone do not meet the bar, she said. "We aren't going to be able to pay for the innovations and keep all the people," she said.Source: Becker's Hospital Review
- In a Vizient survey of 416 senior leaders at 385 U.S. health systems, fielded in April and May and released Nov. 12, 17% of the organizations with the highest returns on AI reported AI-related layoffs; 59% redesigned roles, 56% redeployed staff, 53% reduced overtime and 21% reduced headcount through attrition. Deployments returned a median of 2.5 times and a mean of 3.2 times their cost, figures Vizient said were self-reported estimates that were not audited or independently verified.Sources: Vizient, Becker's Hospital Review
- A Clinical Super Intelligence Roadmap for HHS, CMS and the FDA, posted on the CMS website with a page last updated Nov. 13, says many components of the Medicare Annual Wellness Visit could potentially be delivered via clinical SI tools and that CMS anticipates creating structures for SI-led solutions to participate in Innovation Center models, with capitated models such as LEAD and AHEAD as a payment pathway. It calls for "appropriate human oversight" and "continuous real-world monitoring", says it "does not mandate SI adoption" and sets no deadlines.
- The FBI said in a Nov. 13 alert that cyber actors have used ARTEX, an open-source agentic AI platform that connects to outside language models, in attacks since August, with health care among the targets. John Riggi, the American Hospital Association's national adviser for cybersecurity, said some intrusions progressed from initial targeting to privileged access in under three hours and advised organizations to inventory all internet-facing applications. Google reported Nov. 16 that agentic threat activity had crossed a significant threshold.
Comments on the FDA's draft guidance on robotically assisted surgical devices are due Nov. 24, and RSNA meets in Chicago from Nov. 29 to Dec. 3. Comments on Sen. Bill Cassidy's PATIENT AI Act discussion draft, which would create a Medicare payment pathway for AI tools, are due Dec. 1, and the FDA holds a public workshop on autonomous and remotely operated robotic devices on Dec. 2 and 3. The Federation of State Medical Boards plans to release draft guidance on accountability when AI performs clinical work for public comment in December.
The Medicare Payment Advisory Commission meets Dec. 10 and 11. Comments on CMS's proposed anti-fraud rule, which would require clinicians who bill Medicare Advantage plans to enroll in traditional Medicare, are due 60 days after its publication in the Federal Register. On Jan. 1, Medicare Advantage, Medicaid and some exchange plans must have electronic prior-authorization interfaces running, though a survey found only 1 in 10 payers fully ready.