As AI scraping surges, AI search traffic fails to follow: Report

AI-powered search engines (e.g., OpenAI’s ChatGPT, Perplexity) are failing to drive meaningful traffic to publishers while their web scraping activities increase. That’s one big takeaway from a recent report from TollBit, a platform that says it helps publishers monetize their content.

CTR comparison. Google’s average search click-through rate (CTR) was 8.63%, according to the report. However, the CTR for AI search engines was 0.74% and 0.33% CTR for AI chatbots. That means AI search sends 91% fewer referrals and chatbots send 96% less than traditional search.

Why we care. This is bad news for publishers because it shows AI search won’t replace traditional search traffic. As AI-generated answers replace direct website visits, you should expect to see this trend continue.

By the numbers. AI bot scraping doubled (+117%) between Q3 and Q4 2024. Also:

• The average number of scrapes from AI bots per website for Q4 was 2 million, with another 1.89 million done by hidden AI scrapers.
• 40% more AI bots ignored robots.txt in Q4 than in Q3.
• ChatGPT-User bot activity skyrocketed by 6,767.60%, making it the most aggressive scraper.
• Top AI bots by share of scraping activity:
  • ChatGPT-User (15.6%)
  • Bytespider (ByteDance/TikTok) (12.44%)
  • Meta-ExternalAgent (11.34%)
• PerplexityBot continued sending referrals to sites that had explicitly blocked it, raising concerns about undisclosed scraping.

Context. One company, Chegg, is attempting to sue Google over AI Overviews. Chegg claims Google’s search feature has severely damaged its traffic and revenue.

• Google announced last week an expansion of AI Overviews. It is now starting to show AI Overviews to users who aren’t logged in.

About the data. There’s no methodology section, so it’s not entirely clear how many websites were analyzed, just that it’s based on “all onboarded ToolBit sites in Q4.” Toolbit says it “helps over 500 publisher sites.”

The report. TollBit State of the Bots – Q4 2024 (registration required)

View the full article