Show HN: Katakate – Dozens of VMs per node for safe code exec

Anthropic recently released a sandboxing tool based on bubblewrap (Linux, quite lightweight) and sandbox-exec (macOS). https://github.com/anthropic-experimental/sandbox-runtime

I wonder if nsjails or gVisor may be useful as well. Here's a more comprehensive list of sandboxing solutions: https://github.com/restyler/awesome-sandbox

(sorry I didn't reply in-thread, I'm new to HN, re-posting response here:)

Exactly! The main local requirement is to have hardware virtualization available (e.g. /dev/kvm) but that should be fine on your local linux machine. Won't work in cloud machines or on Mac ARM in current form but maybe if I extend

Are there any such solutions that can adequately protect against side-channel attacks (à la rowhammer, meltdown, spectre, ...)? I mean protecting local file access and network access is pretty easy, but side-channels and VM escaping attacks seem like a bigger concern.

> something that is truly local-first

Hey, we built coderunner[1] exactly for this purpose. It's completely local. We use apple containers for this (which are 1:1 mapped to a lightweight VM).

1. Coderunner - https://github.com/instavm/coderunner

What about this: https://github.com/apple/container

https://rstrict.cloud/ is a CLI built in Rust on top of the Landlock API for the Linux kernel.

It lets you narrow the permission scope of an executable using simple command line wrappers.

Local-first (on Lunix), POSIX shell: https://github.com/sandbox-utils/sandbox-run

No business model short-term. My goal is broad adoption, 100% open-source.

By multi-node I mean so far I only support 1 k8s node, i.e. 1 machine, but soon adding support for multiple. Still, on 20 CPUs I can run +50 VM pods with fractional vCPU limits.

For GPU passthrough: not possible today because I use Firecracker as VMM. On roadmap: Add support for Qemu, then GPU passthrough possible.

Inter-VM networking: it's already possible on single-node: 1 VM = 1 pod. Can have multiple pods per node (have a look at utils/stress-test.sh). Right now I default deny-all ingress for safety (because by default k8s allows inter pod communication), but can make ingress configurable.

Startup time: a second, or a few seconds, depending on which base image (alpine, ubuntu, etc...) and whether you use a before_script or not (what I execute before the network lockdown)

Large artifacts: you can configure resource allocated to a VM pod in the sandbox config and it basically uses k8s resource limits.

Let me know if any other question! Happy to help

Actually you can! After you run "k7 install" you'll have a k3s cluster up and running, with Kata as a runtime class, and Firecracker specified in Kata config. So nothing prevents you from hitting the Kubernetes API! kubectl will work.

Note: I use k3s' internal kubectl and containerd, to avoid messing with your own if you have some already installed. That means you can run commands like "k3s kubectl ..."

And thank you for the compliments on the stack.

If you have any suggestion on how I can make this more friendly UX-wise to your personal usage, I am most interested to hear! And this will shape my roadmap.

It is well known that containers do not provide you safe isolation. It is not their purpose. They share kernel and page cache with the host. Any kernel exploit gives to someone in a container potential root control of the host (see DirtyPipe, DirtyCow). That's why you need VM-level isolation.

It uses Kata with Firecracker which gives you as light of a boot as it gets. Subsecond booting for instance is accessible with a lighter rootfs, which is also on the roadmap (one of the easiest items, actually). The k8s layer doesn't add overhead either compared to any other VM. If you want to compare to bare containers, depending on workload, you could see a 5% overhead due to virtualization. Exact overhead would depend on workload.

Thanks! I'll review Arrakis and come back. E2B is often considered harder to setup and less AI engineers friendly for direct stack contributions, as Katakate is the only alternative fully implemented in Python (core modules, Typer CLI, FastAPI, Python SDK).

Our native K8s support and exposition of K8s API also makes it friendly to devops.

Finally, our deploy/infra stack is lean and tightly fits in a single Ansible playbook, which makes it easy to understand and contribute to, letting you rapidly gain full understanding and ownership of the stack.

That's a config example.

Yes, blocking DNS exfiltration requires DNS filtering at cluster level. This is what will be added with the Cilium integration which is top-3 on the roadmap (top of readme).

DNS resolution is required for basic Kubernetes functionality and hostname resolution within the cluster.

That's said explicitly in several places in the docs: "DNS to CoreDNS allowed"

One thing I could do is make it exposed in config, to allow the user to block all DNS resolutions until Cilium is integrated. LMK if desired!

hey, I work at E2B, anything we can do to improve the setup for you?

Katakate is built on top of Kata, and sets up a stack combining Kubernetes (K3s), Kata, Firecracker, and devmapper snapshotter for thin pool provisioning. Combining these tools together is highly non-trivial and can be a headache for many, especially for AI engs who are often more comfortable with Python workflows. The stack gets deployed with an Ansible playbook. It implements a CLI, API and Python SDK to make it super easy to use. A lot of defense in depth settings are also baked-in so that you don't need to understand those systems at a low level to get a secure setup.

Thanks! Yes: Katakate provides much stronger isolation, since it uses hardware virtualization (via Kata Containers and Firecracker) while gVisor relies purely on software sandboxing in user space.

gVisor isolates containers by intercepting system calls in a user-space kernel, so it can still be vulnerable to sandbox escape via gVisor bugs, though not directly through Linux kernel exploits (since gVisor doesn’t expose the host kernel to the container).

Katakate also provides more than isolation: it offers orchestration through Kubernetes (K3s)

You could create a gVisor RuntimeClass in Kubernetes to orchestrate gVisor sandboxes, but that would require extra setup.

Thanks, I'll review that one too and compare.