- Websites risk losing brand position and authority if they don’t leverage AI
- AI-based discovery may not even see a user navigating to the actual site
- Increased governance over outright bans is the solution
A new Hostinger study of more than 66 billion bot requests across five million websites has revealed how ingrained AI has become in our search habits and how important it is for businesses to get on board with optimizations beyond typical SEO.
The report found that companies that block AI bots to protect their IP may lose control over how AI assistants interpret their content, as well as weaken their brand position.
Hostinger explained the shift in internet discovery, where AI assistants summarize and recommend products and services without users necessarily having to navigate to the website at all.
Is AI replacing traditional search?
“Across all 66.7 billion records, one message stands out: AI crawlers are rapidly increasing their reach, even as AI training bots face increasing resistance from content creators,” Hostinger wrote.
There is a clear reluctance to accept training bots, with GPTBot site coverage dropping from 84% in August to 12% in November. Meta’s ExternalAgent also dropped from 60% to 41%. On the flip side, the crawlers OpenAI’s SearchBot and Applebot increased their coverage.
Despite growing AI crawler activity, traditional search crawler remained fairly flat, leading Hostinger to assume that ai “adds a new decision-making layer above [traditional search]” instead of replacing it — good news for Google, whose discovery spans both traditional search and AI chatbots.
For publishers and content sites, Hostinger explained that site owners may want to increase the AI assistant’s response visibility by using tools like Web2Agent and llms.txt files.
“The real risk for companies is not AI access per se, but losing control over how pricing, positioning and value are presented when decisions are made,” added head of AI Tomas Rasymas.
Looking ahead, the report calls for improved governance over outright bans, for website owners to ensure they are discovered in new AI channels without losing brand position.
Follow TechRadar on Google News and add us as a preferred source to get our expert news, reviews and opinions in your feeds. Be sure to click the Follow button!
And of course you can too follow TechRadar on TikTok for news, reviews, video unboxings, and get regular updates from us on WhatsApp also.
