Kairos: Immutable Distro for K8s at the Edge

It's very different from k3s. With k3s, you have a Linux installation like Debian or Fedora and you install k3s on it. You can SSH into the box, install any other Linux program not running in Kubernetes, etc. It also means that you need to run security updates and all the other stuff that goes along with administering a Linux box.

With Talos, it's just Kubernetes running on the box. There's no SSH or anything. Yes, it's a Linux kernel running, but you don't have a way of running stuff on the box outside of Kubernetes.

For me, Talos is great. If I'm setting up some boxes for K8s, I don't want to have to deal with admin'ing a Linux box. I don't want to login to the box and run some non-K8s service on it. I just want a K8s node and that's what Talos gives me. I think that's also the experience most people want. It's why people pay AWS, GCS, and Azure tons of money to get hosted K8s nodes rather than a Linux box they need to admin.

Not OP, but when we tested it out it was painful to handle usb disks. The reason being that if you have two they get named sda/sdb randomly. We managed to overwrite the usb we were using to install talos since that one was named sda one boot and sdb the next. This lead ut to develop the “pullout technique” when installing…

This mostly only happened because it was a test cluster where we used usb disks, probably not a problem when one properly provisions.

Otherwise it was great! But it does feel akward not booting into an environment where you have a terminal at first

It's an opinionated vertical platform; if you run into an edge case, bug, or functionality you don't like, you are have to open a discussion Github and wait for a new release to fix or change things. The devs are very responsive, but the same as any open source tool, it's their project. It perhaps depends on how much customization you want to do - GPUs and drivers, custom CNI, very specific disk settings. I've had more trouble with bare metal systems with varied hardware vs their supported cloud platforms, which are approved and tested.

I'm pretty positive toward Talos but if you stray from the happy path, by choice or accident, it can become challenging technically. And then you have sunk costs around choosing this platform and how hard it would be to restart from scratch.