Relying on AI chatbots for hospital care is ‘reckless and dangerous’ say human nurses

The next time you’re due for a medical exam you may get a call from someone like Ana: a friendly voice that can help you prepare for your appointment and answer any pressing questions you might have.

With her calm, warm demeanor, Ana has been trained to put patients at ease—like many nurses across the U.S. But unlike them, she is also available to chat 24-7, in multiple languages, from Hindi to Haitian Creole.

That’s because Ana isn’t human, but an artificial intelligence program created by Hippocratic AI, one of a number of new companies offering ways to automate time-consuming tasks usually performed by nurses and medical assistants.

This March 2025 image from the website of artificial intelligence company Xoltar shows a demonstration of one of their avatars for conducting video calls with patients. [Photo: Xoltar via AP]

It’s the most visible sign of AI’s inroads into health care, where hundreds of hospitals are using increasingly sophisticated computer programs to monitor patients’ vital signs, flag emergency situations and trigger step-by-step action plans for care—jobs that were all previously handled by nurses and other health professionals.

Hospitals say AI is helping their nurses work more efficiently while addressing burnout and understaffing. But nursing unions argue that this poorly understood technology is overriding nurses’ expertise and degrading the quality of care patients receive.

“Hospitals have been waiting for the moment when they have something that appears to have enough legitimacy to replace nurses,” said Michelle Mahon of National Nurses United. “The entire ecosystem is designed to automate, de-skill, and ultimately replace caregivers.”

This March 2025 image from the website of artificial intelligence company Xoltar, shows two of of their demonstration avatars for conducting video calls with patients. [Photo: Xoltar via AP]

Mahon’s group, the largest nursing union in the U.S., has helped organize more than 20 demonstrations at hospitals across the country, pushing for the right to have say in how AI can be used—and protection from discipline if nurses decide to disregard automated advice. The group raised new alarms in January when Robert F. Kennedy Jr., the incoming health secretary, suggested AI nurses “as good as any doctor” could help deliver care in rural areas. On Friday, Dr. Mehmet Oz, who’s been nominated to oversee Medicare and Medicaid, said he believes AI can “liberate doctors and nurses from all the paperwork.”

Hippocratic AI initially promoted a rate of $9 an hour for its AI assistants, compared with about $40 an hour for a registered nurse. It has since dropped that language, instead touting its services and seeking to assure customers that they have been carefully tested. The company did not grant requests for an interview.

AI in the hospital can generate false alarms and dangerous advice
Hospitals have been experimenting for years with technology designed to improve care and streamline costs, including sensors, microphones and motion-sensing cameras. Now that data is being linked with electronic medical records and analyzed in an effort to predict medical problems and direct nurses’ care— sometimes before they’ve evaluated the patient themselves.

In this photo provided by National Nurses United, nurses hold a rally in San Francisco on April 22, 2024, to highlight safety concerns about using artificial intelligence in healthcare. [Photo: National Nurses United via AP]

Adam Hart was working in the emergency room at Dignity Health in Henderson, Nevada, when the hospital’s computer system flagged a newly arrived patient for sepsis, a life-threatening reaction to infection. Under the hospital’s protocol, he was supposed to immediately administer a large dose of IV fluids. But after further examination, Hart determined that he was treating a dialysis patient, or someone with kidney failure. Such patients have to be carefully managed to avoid overloading their kidneys with fluid.

Hart raised his concern with the supervising nurse but was told to just follow the standard protocol. Only after a nearby physician intervened did the patient instead begin to receive a slow infusion of IV fluids.

“You need to keep your thinking cap on—that’s why you’re being paid as a nurse,” Hart said. “Turning over our thought processes to these devices is reckless and dangerous.”

Hart and other nurses say they understand the goal of AI: to make it easier for nurses to monitor multiple patients and quickly respond to problems. But the reality is often a barrage of false alarms, sometimes erroneously flagging basic bodily functions—such as a patient having a bowel movement—as an emergency.

“You’re trying to focus on your work but then you’re getting all these distracting alerts that may or may not mean something,” said Melissa Beebe, a cancer nurse at UC Davis Medical Center in Sacramento. “It’s hard to even tell when it’s accurate and when it’s not because there are so many false alarms.”
Can AI help in the hospital?

Even the most sophisticated technology will miss signs that nurses routinely pick up on, such as facial expressions and odors, notes Michelle Collins, dean of Loyola University’s College of Nursing. But people aren’t perfect either.

“It would be foolish to turn our back on this completely,” Collins said. “We should embrace what it can do to augment our care, but we should also be careful it doesn’t replace the human element.”

More than 100,000 nurses left the workforce during the COVID-19 pandemic, according to one estimate, the biggest staffing drop in 40 years. As the U.S. population ages and nurses retire, the U.S. government estimates there will be more than 190,000 new openings for nurses every year through 2032.

Faced with this trend, hospital administrators see AI filling a vital role: not taking over care, but helping nurses and doctors gather information and communicate with patients.

‘Sometimes they are talking to a human and sometimes they’re not’
At the University of Arkansas Medical Sciences in Little Rock, staffers need to make hundreds of calls every week to prepare patients for surgery. Nurses confirm information about prescriptions, heart conditions and other issues—like sleep apnea—that must be carefully reviewed before anesthesia.

The problem: many patients only answer their phones in the evening, usually between dinner and their children’s bedtime.

“So what we need to do is find a way to call several hundred people in a 120-minute window—but I really don’t want to pay my staff overtime to do so,” said Dr. Joseph Sanford, who oversees the center’s health IT.

Since January, the hospital has used an AI assistant from Qventus to contact patients and health providers, send and receive medical records and summarize their contents for human staffers. Qventus says 115 hospitals are using its technology, which aims to boost hospital earnings through quicker surgical turnarounds, fewer cancellations and reduced burnout.

Each call begins with the program identifying itself as an AI assistant.
“We always want to be fully transparent with our patients that sometimes they are talking to a human and sometimes they’re not,” Sanford said.

While companies like Qventus are providing an administrative service, other AI developers see a bigger role for their technology.

Israeli startup Xoltar specializes in humanlike avatars that conduct video calls with patients. The company is working with the Mayo Clinic on an AI assistant that teaches patients cognitive techniques for managing chronic pain. The company is also developing an avatar to help smokers quit. In early testing, patients spend about 14 minutes talking to the program, which can pickup on facial expressions, body language and other cues, according to Xoltar.

Nursing experts who study AI say such programs may work for people who are relatively healthy and proactive about their care. But that’s not most people in the health system.

“It’s the very sick who are taking up the bulk of health care in the U.S. and whether or not chatbots are positioned for those folks is something we really have to consider,” said Roschelle Fritz of the University of California Davis School of Nursing.

---

The Associated Press Health and Science Department receives support from the Howard Hughes Medical Institute’s Science and Educational Media Group and the Robert Wood Johnson Foundation. The AP is solely responsible for all content.

—Matthew Perrone, AP Health Writer

View the full article