Ibm AI ('bob') Downloads and Executes Malware

Yes, Mallory sounds about right.*

———

And before anyone asks for receipts, Eve told me she heard Alice and Bob talking about Gregory Peck's performance.

Unfortunately that opportunistic shortcut is part of the fundamental behavior of modern LLMs, and everybody keeps building hoping the problem will be fixed by some silver-bullet further down the line.

I don't want to sound glib, but one could simply not let an LLM execute arbitrary code without reviewing it first, or only let it execute code inside an isolated environment designed to run untrusted code

the idea of letting an LLM execute code it's dreamt up, with no oversight, in an environment you care about, is absolutely bananas to me

Allowing agent to run wild with any arbitrary shell commands is just plain stupid. This should never happen to begin with.

The number of scenarios in which you have your coding agent retrieving random websites from the internet is very low.

What typically happens is that they use a provider's "web search" API if they need external content, which already pre-processes and summarises all content, so these types of attacks are impossible.

Don't forget: this attack relies on injecting a malicious prompt into a project's README.md that you're actively working on.

I expect that agent LLMs are going to get more and more hardened against prompt injection attacks, but it's hard to get the chance of them working all the way down to zero while still having a useful LLM. So the "solution" is to limit AI privileges and avoid the "lethal trifecta".

Nope, not at all. Non-determinism is what most software developers write. Something to do with profitability and time or something.

Probably good advice for lots of things these days given supply chain attacks targeting build scripts, git, etc.

Users have been trained to do this, as shifting the burden to the user with no way to enforce bounds or even sensible defaults.

E.G. I can guarantee that people will whitelist bwrap, crun, docker, expecting to gain advantage from isolation, while the caller can override all of those protections with arguments.

The reality is that we have trained the public to allow local code execution on their devices to save a few cents on a hamburger, we can’t have it both ways.

Unless you are going to teach everyone that they need to make sure address family 40, openat2(), etc.. are unsafe, users have no way to win right now.

The use case has to either explicitly harden or shift blame.

With Opendesktop, OCI, systemd, and kernel all making locally optimal decisions, the reality is that ephemeral VMs is the only ‘safe’ way to run untrusted code today.

Sandboxes can be better but containers on a workstation (without a machine VM) are purely theatre.

AI sells.

They cant. Why? Because the smartest bear ia smarter than the dumbest human.

So, these AIs are suppose to interface with humans and use nondeterminant language.

That vector will always be exploitable, unless youre talking about AI that no han controls.

This speculative statement is holding way too much of the argument that they are just “beta tools”.

It's a good message for software engineers, who have the context to understand when to take on that liability anyway, but it can lead other job functions into being too trigger-happy on solutions that cause all the same problems with none of the mitigating factors of code.