Guarding My Git Forge Against AI Scrapers

it's called "LLM grooming"

https://thebulletin.org/2025/03/russian-networks-flood-the-i...

Wasn't there a study a while back showing that a small sample of data is good enough to poison an LLM? So I'd say it for sure is possible.

Recognize that a website is a Git repo web interface. Invoke elaborate Git-specific logic. Get the repo link, git clone it, process cloned data, mark for re-indexing, and then keep re-indexing the site itself but only for things that aren't included in the repo itself - like issues and pull request messages.

The scrapers that are designed with effort usually aren't the ones webmasters end up complaining about. The ones that go for quantity over quality are the worst offenders. AI inference-time data intake with no caching whatsoever is the second worst offender.

If you mean scrapers in terms of the bots, it is because they are basically scraping web content via HTTP(S) generally, without specific optimisations using other protocols at all. Depending on the use case intended for the model being trained, your content might not matter at all, but it is easier just to collect it and let it be useless than to optimise it away⁰. For models where your code in git repos is going to be significant for the end use, the web scraping generally proves to be sufficient so any push to write specific optimisations for bots for git repos would come from academic interest rather than an actual need.

If you mean scrapers in terms of the people using them, they are largely akin to “script kiddies” just running someone else's scraper to populate their model.

If by scrapers in terms of people writing them, then the fact that just web scraping is sufficient as mentioned above is likely the significant factor.

> why the scrappers do not do it in a smarter way

A lot of the behaviours seen are easier to reason if you stop considering scrapers (the people using scraper bots) to be intelligent, respectful, caring, people who might give a damn about the network as a whole, or who might care about doing things optimally. Things make more sense if you consider them to be in the same bucket as spammers, who are out for a quick lazy gain for themselves and don't care, or even have the foresight to realise, how much it might inconvenience¹ anyone else.

----

[0] the fact this load might be inconvenient to you is immaterial to the scraper

[1] The ones that do realise that they might cause an inconvenience usually take the view that it is only a small one, and how can the inconvenience little them are imposing really be that significant? They don't think the extra step of considering how many people like them are out there thinking the same. Or they think if other people are doing it, what is the harm in just one more? Or they just take the view “why should I care if getting what I want inconveniences anyone else?”.

I have `REQUIRE_SIGNIN_VIEW=true` and I see nothing but my own traffic on Gitea's logs.

Is it because I'm using a subdomain that doesn't imply there's a Gitea instance behind?

> Enable this to force users to log in to view any page or to use API. It could be set to "expensive" to block anonymous users accessing some pages which consume a lot of resources, for example: block anonymous AI crawlers from accessing repo code pages. The "expensive" mode is experimental and subject to change.

Forgejo doesn't seem to have copied that feature yet

When scraping was mainly used to build things like search indexes, which are ultimately mutually beneficial to both the website owner and the search engine, and the scrapers were not abusive, nobody really had a problem.

But, for generative AI training and access, with scrapers that DDoS everything in sight, and which ultimately cause visits to the websites to fall significantly and merely return a mangled copy of its content back to the user, scraping is a bad thing. It also doesn't help that the generative AI companies haven't paid most people for their training data.

But for some reason corporations don't want that, I guess they want to be allowed to just take from the commons and give nothing in return :/

I recall that bot farms use pre-paid SIM cards for their data connections so that their traffic comes from a good residential ASN.

No client compromise required, it's a networking abuse that gives you good reputation of you use mobile data.

But yes, selling botnets made of compromised devices is also a thing.

Much more likely are those companies that pay people (or trick people) into running proxies on their home networks to help with giant scrapping projects what want to rotate through thousands of "real" IPs.

If you're just blocking their IP, e.g. "we've had to block your access because your device, or a device on your network, appears to be sending abusive traffic to us. this sometimes happens because an app on your device is doing this without your knowledge".

If you've had to block a whole AS, you could name and shame them, e.g. "we've had to block your access because your internet provider, <name>, appears to be sending abusive traffic to us. consider letting them know, or moving providers".

Of course, this would almost certainly be commercially stupid. It would also only really work if everyone decided to do this, which realistically will never happen.

no

for many reasons

My company runs our VPN from our datacenter (although we have our own IP block, which hopefully doesn’t get blocked)

Our public web dataset goes back to 2008, and is widely used by academia and startups.