Walling Off the Open Internet to Stop AI May End Up Breaking Everything Else
Posted4 months agoActive4 months ago
techdirt.comTechstory
heatedmixed
Debate
70/100
Artificial IntelligenceInternet FreedomWeb Scraping
Key topics
Artificial Intelligence
Internet Freedom
Web Scraping
The article discusses how efforts to restrict AI web scraping may end up fragmenting the internet, and commenters debate the ethics and consequences of such measures.
Snapshot generated from the HN discussion
Discussion Activity
Moderate engagementFirst comment
N/A
Peak period
6
0-1h
Avg / period
2.5
Key moments
- 01Story posted
Sep 8, 2025 at 2:05 PM EDT
4 months ago
Step 01 - 02First comment
Sep 8, 2025 at 2:05 PM EDT
0s after posting
Step 02 - 03Peak activity
6 comments in 0-1h
Hottest window of the conversation
Step 03 - 04Latest activity
Sep 9, 2025 at 3:15 AM EDT
4 months ago
Step 04
Generating AI Summary...
Analyzing up to 500 comments to identify key contributors and discussion patterns
ID: 45171677Type: storyLast synced: 11/20/2025, 5:57:30 PM
Want the full context?
Jump to the original sources
Read the primary article or dive into the live Hacker News thread when you're ready.
That's not how boundaries work.
just like signs such as no trespassing: active shooting range; or apiary[bee] yard; or open shaft mines, means that exactly, and in my region, signage is sufficient no gate or fence required.
every so often someone thinks they are too important for that to apply and they find out the signage is no joke when they encounter the target line, and backstop being hit from somewhere uprange; or violate the space of a bee hive; or fall into a shaft. some even have the nerve to complain this is a threat.
robots.txt == no trespassing == stay out unless invited
An AI that is randomly guessing passwords to an authentication/login endpoint is clearly malicious.
I think it's more like this:
Is it unethical to lock/obstruct the public sidewalk in front of your house because it blocks people who only want to pass by and look at the house?
the former tier will shortly be 100% AI slop, so the latter
personally I'd be willing to sacrifice the internet if it renders generative AI impotent
there are many possible futures that are bleaker than a world without internet
> We're Walling Off The Open Internet To Stop AI—And It May End Up Breaking Everything Else
I do believe in and love the open web. But the AI scrapers need to change, not me! They're the ones with the money and the 10x engineers (apparently). Read my sitemap and crawl my content from one IP, don't just haphazardly crawl and click every damn link and search term from hundreds or thousands of IPs every day!