Skip to content
Hot storiesContinuously updated

Trakis发布网站屏蔽AI爬虫现状调研

1 reports1 reporting sourcesUpdated 6 hours ago

Start here

AI overview

2026年11月4日，Trakis Research 发布了对 13,938 个网站 robots.txt 文件的调查统计结果。该调研探讨了拦截 AI 训练的站点是否同时屏蔽回答类爬虫。数据显示，全网拦截 AI 训练爬虫的频率约为拦截回答与搜索爬虫的三倍；此外，在按名称屏蔽 OpenAI 旗下训练爬虫 GPTBot 的网站中，有 17.1% 同时也屏蔽了用于搜索和回答的 OAI-SearchBot。

AI-generated from reports · Updated 5 hours ago

Latest developmentsNov 4 21:16
Trakis调查显示全网拦截AI训练爬虫频率约为搜索爬虫的三倍。

Report timeline

Follow the reports to see different sides of the story.

Nov 4
  1. Hacker News · AISelected
    Trakis Research Analyzes 13,938 Websites: Do Sites Blocking AI Training Also Block Answer Crawlers?

    A Trakis Research study analyzing robots.txt files across 13,938 websites shows that web-wide blocking of AI training crawlers occurs roughly three times as often as that of answer and search crawlers, with 17.1% of sites blocking GPTBot by name also blocking OAI-SearchBot.

Story heat trend

Current heat 9·Comparable peak 10（Nov 4 22:00）·Comparable change, last 24 hours–

02.557.510Nov4Nov5Nov5Nov5

Trends compare only the same parties observed continuously, so the scope can be narrower than current heat counts. Move the pointer or tap the chart to see hourly heat; use the left and right arrow keys on a keyboard.