New York lawmakers want AI chatbots to stop pretending to be doctors or lawyers

New York is the latest state to consider a bill that would prohibit AI chatbots from dispensing advice that licensed professionals would normally give, such as medical or legal advice. The bill would also allow people who believe they were harmed by such advice to sue the operator of the chatbot.

Senate Bill S7263, introduced by Democratic state Senator Kristen Gonzalez, passed out of a technology committee on a 6–0 vote last week and now advances to a reading on the floor of the Senate. Interestingly, the bill requires operators to clearly label their chatbots as AI, but stipulates that such a label isn’t enough to shield them from lawsuits under the statute.

The proposal reflects a growing shift in how policymakers are thinking about AI. While early efforts focused mostly on transparency, lawmakers are beginning to explore something arguably more consequential: whether companies should be legally liable when AI systems give advice that causes real-world harm.

The bill applies to chatbots that give advice in the fields of medicine, law, dentistry, veterinary medicine, physical therapy, pharmacy, nursing, podiatry, optometry, engineering, land surveying, geology, architecture, psychology, and social work.

New York isn’t acting alone. Other states have passed or are considering similar laws, though with varying scopes and enforcement methods, and they tend to focus primarily on healthcare:

• California’s AB 489, enacted in 2025, does something similar but with a narrower scope, targeting AI systems that misrepresent their information as coming from licensed healthcare professionals. But AB 489 relies on state healthcare boards for enforcement and doesn’t provide a private right of action (civil suit) for legal recourse.
• A new Nevada law, AB 406, which went into effect last July, prohibits the advertising and operation of AI systems designed to dispense professional mental and behavioral healthcare therapy. The law also limits how licensed professionals can use AI in their practices.
• Last August, Illinois passed HB 1806, which prohibits licensed therapists in the state from using AI to make treatment decisions or communicate with clients. The law also prohibits tech companies from advertising or offering AI-powered therapy services in the state without the involvement of a licensed professional.
• Utah passed a similar law, HB 452, that puts restrictions and disclosure requirements on chatbots that appear to offer an alternative to human mental health therapy or advice. The law went into effect in 2025.

Professional medical groups have also begun weighing in on the risks. The American Medical Association doesn’t call for a ban on AI chatbots dispensing health information, but it worries that consumer advice from LLMs might be false or misleading. “Notably, tools such as ChatGPT have shown a not-uncommon tendency to falsify references cited in response to these queries,” the AMA says in a policy paper, adding that AI tools have demonstrated the ability to generate fraudulent scientific or medical literature to support health advice.

An especially sensitive area is mental health advice. Mental health advice is a particularly sensitive area, perhaps because many chatbot users, especially younger ones, use AI as a counselor or therapist. A 2025 JAMA Network study found that 13% of all respondents used chatbots for mental health advice, with 22% of those ages 18 to 21 doing so.

AI companies have recognized a large and growing addressable market for AI mental health chatbots, in part because many consumers struggle to find an affordable therapist. For some consumers, the argument goes, accessing mental health advice from a chatbot is better than receiving none at all.

The American Psychological Association says chatbots could actually be worse than nothing at all. That’s because of AI models’ tendency to relate to humans in a sycophantic way. The group said in a presentation to the Federal Trade Commission that “A.I. chatbots ‘masquerading’ as therapists, but programmed to reinforce, rather than challenge a user’s thinking, could drive vulnerable people to harm themselves or others.”

Senator Gonzalez will likely have to explain why her S7263 bill targets chatbots as a source of legal or medical information but not traditional search engines. She might cite a 2025 experimental study showing that people “over-trust” AI. (She declined to comment for this story.) The researchers write that AI chatbots often sound more convincing and trustworthy than search engine results—even when they’re wrong, off-topic, or lacking context. The study also found that participants couldn’t reliably distinguish AI responses from those from real doctors, and often rated AI responses as more trustworthy and complete.

Another study by Oxford and MLCommons involving nearly 1,300 participants found that using AI models to evaluate and analyze symptom scenarios led to health decisions (“should I take some aspirin? Should I go to the ER?”) that were no better than decisions based on personal knowledge or information gleaned from traditional internet search. The study also found that users often don’t know what details the AI needs to generate an answer, and that the AI’s outputs often blend correct and incorrect recommendations.

The AMA says it would like to see the Federal Trade Commission regulate AI chatbots that dispense health information, but believes the agency currently lacks the resources to take on that role. Instead, it calls on the tech companies behind the chatbots to continually review the accuracy of their models and give users an easy way to report when a chatbot outputs inaccurate health information.

View the full article