Hacker News · AI· researchaiexe·· 6 hours agoSelectedAI score66
Trakis Research Analyzes 13,938 Websites: Do Sites Blocking AI Training Also Block Answer Crawlers?
Block training, keep answers? What 13,938 sites ask of AI bots
AI guide
A Trakis Research study analyzing robots.txt files across 13,938 websites shows that web-wide blocking of AI training crawlers occurs roughly three times as often as that of answer and search crawlers, with 17.1% of sites blocking GPTBot by name also blocking OAI-SearchBot.
Why it matters
The findings highlight how websites differentiate between blocking AI training crawlers and search/answer crawlers, helping webmasters check whether their robots.txt directives inadvertently suppress citations in generative search engines.
Source:Hacker News · AI · trakis.io