Analyzing 5,818 Publishers' Robots.txt Files
Posted3 months ago
newoldweb.comTechstory
calmneutral
Debate
0/100
Robots.txtAINews OrganizationsWeb Scraping
Key topics
Robots.txt
AI
News Organizations
Web Scraping
The article analyzes the robots.txt files of 5,818 publishers, revealing that most non-profit news organizations allow AI bots, with OpenAI being the most commonly blocked.
Snapshot generated from the HN discussion
Discussion Activity
No activity data yet
We're still syncing comments from Hacker News.
Generating AI Summary...
Analyzing up to 500 comments to identify key contributors and discussion patterns
ID: 45633975Type: storyLast synced: 11/17/2025, 9:05:13 AM
Want the full context?
Jump to the original sources
Read the primary article or dive into the live Hacker News thread when you're ready.
Discussion hasn't started yet.