- AI bots and scrapers may soon overtake human users on the Internet
- Publishers struggle with dwindling click-through rates
- robots.txt instructions are largely ignored
New data has claimed that AI bots are rapidly taking over web traffic, with Tollbit data showing that there was one new AI bot visit for every 31 human visits in the final months of 2025, up from 1:200 at the start of 2025.
At the same time, and with humans interacting more directly with AI, human visits fell by around 5% between Q3 and Q4 2025.
And it’s clear that human AI use is responsible for this shift – while training crawls fell by about 15% between Q2 and Q4 2025, RAG bots increased by about 33%, and AI search indexers increased by about 59%.
The way we access the Internet is changing fundamentally
OpenAI leads the way in terms of scraping – its RAG bot ‘ChatGPT-User’ was about five times more active than the second most active bot of Meta, and about 16 times higher than Perplexity’s agent.
In terms of user behavior, a separate study cited in the report found that more than a third (37%) of active AI users now initiate searches with artificial intelligence (such as ChatGPT or Gemini) instead of traditional search.
More alarmingly, the analysis showed that robots.txt, a set of instructions to tell automated bots which parts of the page they can and cannot crawl, was ignored about 30% of the time on average and up to 42% of the time by ChatGPT-User.
Since robots.txt is unenforceable and entirely dependent on goodwill, the instructions have been deemed de facto obsolete.
And that’s not the only bad news for publishers struggling on the traffic front. Sites without direct AI license agreements saw click-through rates (CTR) drop about 3x between Q2 and Q4 2025. Even those with AI license agreements are not immune, with CTR rates also falling.
With all this at stake, Tollbit is calling for regulators to step in to establish acceptable AI patterns and protect intellectual property.
But with AI nowhere to be found, it’s clear that the web is undergoing a major shift, and website owners should prepare for AI bots to become their primary readers.
Follow TechRadar on Google News and add us as a preferred source to get our expert news, reviews and opinions in your feeds. Be sure to click the Follow button!
And of course you can too follow TechRadar on TikTok for news, reviews, video unboxings, and get regular updates from us on WhatsApp also.
