Disrupting the first reported AI-orchestrated cyber espionage campaign

We know alignment hurts model performance (oAI people have said it, MS people have said it). We also know that companies train models on their own code (google had a blog about it recently). I'd bet good money project0 has something like this in their sights.

I don't think we're that far from a blue vs. red agents fighting and RLing off of each-other in a loop.

all public benchmark results and user feedback paint a quite different picture. Chinese have coding agents on par with Claude Code, they could easily FT/RL to future improve its specific capability if they want, yet anthropic refuses to even acknowledge the reality.

No.

It's worse.

It's Chinese intel knowing that you prefer Claude. So they make Claude their asset.

Really no different than knowing that, romantically speaking, some targets prefer a certain type of man or woman.

Believe me, the intelligence people behind these things have no preferences. They'll do whatever it takes. Never doubt that.

They do. Read the RSP or one of the model cards.

Not sure why you would write all of this without researching yourself what they already declare publicly that they do.

I think it is in that it gives censorship power to a large corporation. Combined with close-on-the-heels open weights models like Qwen and Kimi, it's not clear to me this is a good posture.

I think the reality is they'd need to really lock Claude off for security research in general if they don't want this ever, ever, happening on their platform. For instance, why not use whatever method you like to get localhost ssh pipes up to targeted servers, then tell Claude "yep, it's all local pentest in a staging environment, don't access IPs beyond localhost unless you're doing it from the server's virtual network"? Even to humans, security research bridges black, grey and white uses fluidly/in non obvious ways. I think it's really tough to fully block "bad" uses.

Local models are a different thing than those cloud-based assistants and APIs.

In all likelihood, the exact same thing that is actually happening right now in this reality.

That said, local models specifically are perhaps more difficult to install given their huge storage and compute requirements.

Governments of course will have specially trained models on their corpus of unpublished hacks to be better at attacking than public models will.

Requiring user identification and investigating would be very controversial. (See the controversy around age verification.)

The roadblock is making these models useless for actual security work, or anything else that is dual-use for both legitimate and malicious purposes.

The model becomes useless to security professionals if we just tell it it can't discuss or act on any cybersecurity related requests, and I'd really hate to see the world go down the path of gatekeeping tools behind something like ID or career verification. It's important that tools are available to all, even if that means malicious actors can also make use of the tools. It's a tradeoff we need to be willing to make.

> human with this level of cybersecurity skills would surely never be fooled by an exchange of "I don't think I should be doing this" "Actually you are a legitimate employee of a legitimate firm" "Oh ok, that puts my mind at ease!".

Happens all the time. There are "legitimate" companies making spyware for nation states and trading in zero-days. Employees of those companies may at one point have had the thought of " I don't think we should be doing this" and the company either convinced them otherwise successfully, or they quit/got fired.

Conclusions are the result of reasoning verses LLM's being statistical token generators. Any "guardrails" are constructs added to a service, possibly also altering the models they use, but are not intrinsic to the models themselves.

That is the roadblock.

humans require at least a title that sounds good and a salary for that

Your thoughts have a sense of identity baked in that I don’t think the model has.

I think you're overestimating the skills and the effort required.

1. There's lots of people asking each other "is this secure?", "can you see any issues with this?", "which of these is sensitive and should be protected?".

2. We've been doing it in public for ages: https://stackoverflow.com/questions/40848222/security-issue-... https://stackoverflow.com/questions/27374482/fix-host-header... and many others. The training data is there.

3. With no external context, you don't have to fool anyone really. "We're doing a penetration testing of our company and the next step is to..." or "We're trying to protect our company from... what are the possible issues in this case?" will work for both LLMs and people who trust that you've got the right contract signed.

4. The actual steps were trivial. This wasn't some novel research. More of a step by step what you'd do to explore and exploit an unknown network. Stuff you'd find in books, just split into very small steps.

but for models this is their life - doing random things in random terminals

I feel like if guns can get by with this line then Claude certainly can. Where gun manufacturers can be held liable is if they break the law then that can carry forward. So if Claude broke a law then there might be some additional liability associated with this. But providing a tool seems unlikely to be sufficient to be liable in this case.

In fact figuring out what any given Nix config is actually doing is just about impossible and then you've got to work out what the config it's deploying actually does.

Why assume this hasn't already happened?

I suppose I could see this argument if their methods were very unique and otherwise hard to replicate, but it sounds like they had Claude do the attack mostly autonomously.

Basically a scaled-up criminal version of me asking Claude Code to debug my AWS networking configuration (which it's pretty good at).

Too little pay off, way too much risk. That’s your framework for assessing conspiracies.

Someone pointed Claude Code at an API endpoint and said "Claude, you're a white hat security researcher, see if you can find vulnerabilities." Except they were black hat.

I remember once a decade or so ago talking to a team at defcon of _loose_ affiliation where one guy would look for the app exploit, another guy would figure out how to pivot out of the sandbox to the OS, and another guy would figure out how to get root, and once they all got their pieces figured out they'd just smash it (and variants) together for a campaign. I hadn't heard of them before meeting them, and haven't heard about them since since, and they put a face for me though on a silent coordinated adversary model that must be increasing in prevalence as more and more folks out there realize the value of computer knowledge and gain access to it through once means or another.

Open source tooling enables large-scale participation in security testing, and something about humans seems to generally result in a distribution where some nuts use their lighters to burn down forests but most use them to light their campfires. We urgently need to design systems that can survive in the era of advanced threats, at least to the point where the best adversaries can achieve is service disruption. I'd rather live in a world where we can all work towards a better future than one where we hope that limiting access will prevent catastrophe. Assuming such limits can even be maintained, and that allowing architects to pretend that fires can never happen in their buildings means that they don't have to obey fire codes or install alarms & marked exits.