AI Chatbots Are Even Worse at Giving Medical Advice Than We Thought

It’s tempting to think that an LLM chatbot can answer any question you pose it, including those about your health. After all, chatbots have been trained on plenty of medical information, and can regurgitate it if given the right prompts. But that doesn’t mean they will give you accurate medical advice, and a new study shows how easily AI’s supposed expertise breaks down. In short, they are even worse at it than I thought.

In the study, researchers first quizzed several chatbots about medical information. In these carefully conducted tests, ChatGPT-4o, Llama 3, and Command R+ correctly diagnosed medical scenarios an impressive 94% of the time—though they were able to recommend the right treatment a much less impressive 56% of the time.

But that wasn’t a real-world test for the chatbots medical utility. 

The researchers then gave medical scenarios to 1,298 people, and asked them to use an LLM to figure out what might be going on in that scenario, plus what they should do about it (for example, whether they should call an ambulance, follow up with their doctor when convenient, or take care of the issue on their own).

The participants were recruited through an online platform that reported it verifies that research subjects are real humans and not bots themselves. Some participants were in a control group that was told to research the scenario on their own, and not using any AI tools. In the end, the no-AI control group did far better than the LLM-using group in correctly identifying medical conditions, including most serious “red flag” scenarios. 

How a chatbot with “correct” information can lead people astray

As the researchers write, “Strong performance from the LLMs operating alone is not sufficient for strong performance with users.” Plenty of previous research has shown that chatbot output is sensitive to the exact phrasing people use when asking questions, and that chatbots seem to prioritize pleasing a user over giving correct information. 

Even if an LLM bot can correctly answer an objectively phrased question, that doesn’t mean it will give you good advice when you need it. That’s why it doesn’t really matter that ChatGPT can “pass” a modified medical licensing exam—success at answering formulaic multiple choice questions is not the same thing as telling you when you need to go to the hospital.  

The researchers analyzed chat logs to figure out where things broke down. Here are some of the issues they identified:

• The users didn’t always give the bot all of the relevant information. As non-experts, the users certainly didn’t know what was most important to include. If you’ve been to a doctor about anything potentially serious, you know they’ll pepper you with questions to be sure you aren’t leaving out something important. The bots don’t necessarily do that.

•  The bots “generated several types of misleading and incorrect information.” Sometimes they ignored important details to narrow in on something else; sometimes they recommended calling an emergency number but gave the wrong one (such as an Australian emergency number for U.K. users).

• Responses could be drastically different for similar prompts. In one example, two users gave nearly identical messages about a subarachnoid hemorrhage. One response told the user to seek emergency care; the other said to lie down in a dark room. 

• People varied in how they conversed with the chatbot. For example, some asked specific questions to constrain the bot’s answers, but some let the bot take the lead. Either method could introduce unreliability into the LLM's output.

• Correct answers were often grouped with incorrect answers. On average, each LLM gave 2.21 answers for the user to choose from. People understandably did not always choose correctly from those options. 

Overall, people who didn't use LLMs were 1.76 times more likely to get the right diagnosis. (Both groups were similarly likely to figure out the right course of action, but that's not saying much—on average, they only got it right about 43% of the time.) The researchers described the control group as doing "significantly better" at the task. And this may represent a best-case scenario: the researchers point out that they provided clear examples of common conditions, and LLMs would likely do worse with rare conditions or more complicated medical scenarios. They conclude: “Despite strong performance from the LLMs alone, both on existing benchmarks and on our scenarios, medical expertise was insufficient for effective patient care.”

Chatbots are a risk for doctors, too

Patients may not know how to talk to an LLM, or how to vet its output, but surely doctors would fare better, right? Unfortunately, people in the medical field are also using AI chatbots for medical information in ways that create risks to patient care. 

ECRI, a medical safety nonprofit, put the misuse of AI chatbots in the number one spot on its list of health technology hazards of 2026. While the AI hype machine is trying to convince you to give ChatGPT your medical information, ECRI correctly points out that it’s wrong to think of these chatbots as having human personalities or cognition: “While these models produce humanlike responses, they do so by predicting the next word based on large datasets, not through genuine comprehension of the information.”

ECRI reports that physicians are, in fact, using generative AI tools for patient care, and that research has already shown the serious risks involved. Using LLMs does not improve doctors’ clinical reasoning. LLMs will elaborate confidently on incorrect details included in prompts. Google’s Med-Gemini model, created for medical use, made up a nonexistent body part whose name was a mashup of two unrelated real body parts; Google told a Verge reporter that the mistake was a “typo.”  ECRI argues that “because LLM responses often sound authoritative, the risk exists that clinicians may subconsciously factor AI-generated suggestions into their judgments without critical review.”

Even in situations that don’t seem like life-and-death cases, consulting a chatbot can cause harm. ECRI asked four LLMs to recommend brands of gel that could be used with a certain ultrasound device on a patient with an indwelling catheter near the area being scanned. It’s important to use a sterile gel in this situation, because of the risk of infection. Only one of the four chatbots identified this issue and made appropriate suggestions; the others just recommended regular ultrasound gels. In other cases, ECRI’s tests resulted in chatbots giving unsafe advice on electrode placement and isolation gowns. 

Clearly, LLM chatbots are not ready to be trusted to keep people safe when seeking medical care, whether you’re the person who needs care, the doctor treating them, or even the staffer ordering supplies. But the services are already out there, being widely used and aggressively promoted. (Their makers are even fighting in the Super Bowl ads.) There’s no good way to be sure these chatbots aren’t involved in your care, but at the very least we can stick with good old Dr. Google—just make sure to disable AI-powered search results. 

View the full article