Back to Home11/15/2025, 11:30:04 PM

Blocking LLM crawlers without JavaScript

todsacerdoti
190 points
90 comments

Mood

thoughtful

Sentiment

positive

Category

tech

Key topics

LLM

web scraping

crawlers

website management

Debate intensity60/100

The article discusses a method to block LLM crawlers without using JavaScript, providing a solution for website owners to manage unwanted traffic.

Snapshot generated from the HN discussion

Discussion Activity

Light discussion

First comment

1d

Peak period

2

Day 2

Avg / period

2

Comment distribution2 data points

Based on 2 loaded comments

Key moments

  1. 01Story posted

    11/15/2025, 11:30:04 PM

    3d ago

    Step 01
  2. 02First comment

    11/17/2025, 1:09:14 AM

    1d after posting

    Step 02
  3. 03Peak activity

    2 comments in Day 2

    Hottest window of the conversation

    Step 03
  4. 04Latest activity

    11/17/2025, 1:34:08 AM

    2d ago

    Step 04

Generating AI Summary...

Analyzing up to 500 comments to identify key contributors and discussion patterns

Discussion (90 comments)
Showing 2 comments of 90
snehesht
2d ago
1 reply
Isn’t this easy for LLMs to avoid by passing an instruction to ignore any hidden links ?
krackers
2d ago
Companies mass crawling don't use LLMs for crawling itself, that would be too expensive.

88 more comments available on Hacker News

View full discussion on Hacker News
ID: 45941441Type: storyLast synced: 11/16/2025, 9:42:57 PM

Want the full context?

Jump to the original sources

Read the primary article or dive into the live Hacker News thread when you're ready.

Read ArticleView on HN