Most Publishers Allow AI Bots to Crawl Their Sites
Posted3 months ago
newoldweb.comTechstory
calmneutral
Debate
0/100
Artificial IntelligenceWeb CrawlingPublishing
Key topics
Artificial Intelligence
Web Crawling
Publishing
An analysis of 5818 publishers' robots.txt files found that most allow AI bots to crawl their sites, with non-profit news organizations being more permissive and OpenAI being the most commonly blocked AI bot.
Snapshot generated from the HN discussion
Discussion Activity
No activity data yet
We're still syncing comments from Hacker News.
Generating AI Summary...
Analyzing up to 500 comments to identify key contributors and discussion patterns
ID: 45595840Type: storyLast synced: 11/17/2025, 10:08:10 AM
Want the full context?
Jump to the original sources
Read the primary article or dive into the live Hacker News thread when you're ready.
Discussion hasn't started yet.