This BBC Study Shows How Inaccurate AI News Summaries Actually Are

It turns out that getting your news from robots playing telephone with actual sources might not be the best idea. In a BBC study of OpenAI, Google Gemini, Microsoft Copilot, and Perplexity’s news prowess, the news organization found that “51% of all AI answers” about news topics had “significant issues of some form.”

The study involved asking each bot to answer 100 questions about the news, using BBC sources when available, with their answers then being rated by “journalists who were relevant experts in the subject of the article.”

A few examples of issues include Gemini suggesting that the UK’s NHS (National Health Service) does not recommend vaping as a method for quitting smoking (it does), as well as ChatGPT and Copilot saying politicians who had left office were actually still serving their terms. More concerning, Perplexity misrepresented a BBC story on Iran and Israel, attributing viewpoints to the author and his sources that the article does not share.

Regarding its own articles specifically, the BBC says 19% of AI summaries introduced these kinds of factual errors, hallucinating false statements, numbers, and dates. Additionally, 13% of direct quotes were “either altered from the original source or not present in the article cited.”

Inaccuracies were not fully distributed between the bots, although this might come as cold comfort given that none performed especially well either. 

“Microsoft's Copilot and Google's Gemini had more significant issues than OpenAI's ChatGPT and Perplexity,” the BBC says, but on the flip side, Perplexity and ChatGPT each still had issues with more than 40% of responses.

In a blog, BBC CEO Deborah Turness had harsh words for the tested companies, saying that while AI offers “endless opportunities,” current implementations of it are “playing with fire.”

"We live in troubled times,” Turness wrote. “How long will it be before an AI-distorted headline causes significant real world harm?"

The study is not the first time the BBC has called out AI news summaries, as its prior reporting arguably convinced Apple to shut down its own AI news summaries just last month.

Journalists have also previously butted heads with Perplexity over copyright concerns, with Wired accusing the bot of bypassing paywalls and the New York Times sending the company a cease-and-desist letter. News Corp, which owns the New York Post and The Wall Street Journals, went a step further, and is currently suing Perplexity.

To conduct its tests, the BBC temporarily lifted restrictions preventing AI from accessing its sites, but has since reinstated them. Regardless of these blocks and Turness’ harsh words, however, the news organization is not against AI as a rule.

“We want AI companies to hear our concerns and work constructively with us,” the BBC study states. “We want to understand how they will rectify the issues we have identified and discuss the right long-term approach to ensuring accuracy and trustworthiness in AI assistants. We are willing to work closely with them to do this.”

View the full article