Kubernetes Is Your Private Cloud

IMO an article like this shouldn't just make the claim - it should show how to do it at the home lab level.

“Everything You Expect from a Cloud, Running on Your Terms“*

Except you own ops, management, extension, interoperability, access, security, scalability, redundancy… words cannot express how ridiculous all of the koober propaganda is

Kubernetes is powerful, yes. it is also a feckless rats nest of bolt-ons and ride-alongs. its sharepoint levels of byzantine tuning so complex that, like sharepoint, it comes with its own bespoke administrators that often have little or no knowledge of basic networking or operating systems --only kubernetes--.

- Upgrading a kubernetes cluster may as well be an olympic sport. its so draconian most best practice documentation insists you build a second cluster for AB deployment.

- load balancers come in half a dozen flavours, with the default options bolted at the hip to the cloud cartel. MetalLB is an option, but your admin doesnt understand subnets let alone BGP.

- It is infested with the cult of immutability. pod not working? destroy it. network traffic acting up? destroy the node. container not working? time to destroy it. cluster down? rebuilt the entire thing. At no point does the "devops practitioner" stop to consider why or how a thing of kubernetes has betrayed them. it is assumed you have a football field of fresh bare metal to reinitialize everything onto at a moments notice, failure modes be damned.

what your company likely needs is some implementation of libvirtd or proxmox. run your workloads on rootless podman or (god forbid) deploy to a single VM.

Managed Ceph in the past. I cannot comprehend someone putting up with the headache that is Ceph in their home lab. To each their own!

Can't wait for k8s hype to go the way of microservices.

As someone who self hosted bare metal Kubernetes on my own rack, it's a lot of work to get it set up. We used RedHat Openshift which has a pretty good solution out of the box, but the learning curve was relatively high.

That being said, once it was set up, there was not a lot of maintenance. Kubernetes is quire resilient when set up properly, and the cost savings were significant.

Been using Nomad and different sorts of K8S for the last 6+ years at home/work. Nomad is easier to bootstrap, lighter on resources and so so so much easier wrap your head around. Just Nomad + NFS server, from my perspective, is a perfect start for a homelab/small project. You can add complexity to it as you go. It is a real joy work with once you after a day of tinkering. Want to run on windows? Sure. BSD, there is a driver for it. Don't want Docker? There is Podman driver. OCI sucks? Just run binaries without isolation. Need VM's? You can switch from Proxmox to purely Nomad setup with Qemu driver with a bit of sweat. Illumos zones on OmniOS? weird, but with quite a bit of time, but there was a repo on github, just need to build the binary with the patch.

And while k8s can do all the same things and much more with a bit of trying, but it requires a mission control the second you add a second developer, you will have built-in primitives that will compete all the time with the ones you bolt-on, etc etc. Nomad feels much more opinionated and in a good way.

Nomad is one of those things that gets you 90% of the way with 20% of effort, and only then if you need something, you can add things to it. K8S is great, way more flexible, there are managed options out there, massive ecosystem, but it always feels like out of the box you need to glue 5 different tools to it, just get it going.

Also Incus. Stephane Graeber is doing lords work by sticking to his thing. That's also super fun to mess with.

As long as you have someone to babysit your cluster.

Counter Argument to "HA HA, Kubernetes sucks": https://www.macchaffee.com/blog/2024/you-have-built-a-kubern...

Kubernetes is a rat nest and I long hope for Kubernetes be simpler (Who needs this Gateway API?) but Devs keep building crazy and crazy solutions so we have to pivot to keep up.

No, what I need is NixOS, a configuration with a language that's a bit hard to digest but effective, which I can build and read, allowing me to replicate my infrastructure, create custom ISOs etc, in an almost totally automatic and manageable way on domestic hardware at a domestic scale.

What's needed isn't a rambling YAML and immense resource consumption; it's IaC, built-in to the system, that can do what's necessary not for an IT giant that lives off others' services run in-house, but for me, a private citizen with just a few of my own services, little time to manage them, a need to quickly replicate the infrastructure because I don't have infinite data centers, so if the homeserver dies, I need to buy another cheap desktop and set it up, restoring it on the fly. So if I'm offline for a few hours, nothing happens, but hardware costs money, so I need to use it well, and so on.

The giants' solutions are not one-size-fits-all.

Now you have two problems: kubernetes and your private cloud. The second being that you decided you needed "cloud" to start with.

It'd be easier to ask Kubernetes fans what Kubernetes isn't at this point.

great post but maybe missing some things, like configuring load balancers with metallb, CNI's like Calico, how you can get your own address space from ARIN for IPv6, and API gateways

There is a fundamental misunderstanding people have about technology that you only get from operating and maintaining it. There's what something can do, and then there's actually doing it.

K8s is complicated as hell to learn to use. Its learning curve is very shallow. Yes, you can get a "hello world" running quickly, but that is not the benchmark for whether you actually understand what's going on, or how to make it do what you need.

But once you do learn it thoroughly, it's ridiculously fast to ramp up on it and use it for very complex things, very quickly. As a developer medium, as an operational medium, it accelerates the kind of modern practices (that for some reason most people still don't know about?) that can produce a lot of value.

But that's if someone else is building and maintaining the underlying "guts" of it for you. If you try to build it from scratch and maintain it on bare metal, suddenly it's incredibly complicated again with another shallow learning curve and a lot of effort. And that effort continues, because they keep changing the fucking thing every 6-12 months...

It's like learning to drive a car or ride a bike. At first it's difficult and time-consuming. But then you get it, and you can ride around pretty easily, and it can take you far. However, that does not mean you understand car/bike mechanics, or can take it apart and rebuild it when something breaks. So be prepared to pay someone for that maintenance, or be prepared to become a mechanic (and do that work while also doing your day job). This analogy is stretched thinner by the fact that nobody's constantly changing how your car works...

> because everyone else seemed to do it

This is not a good reason to have done it. To me this means that the expectations and outcomes were flawed as they are solving a problem that shouldn't have existed. I can't really agree with the sentiment or overview of this post

Speaking as someone who is deploying HA Kubernetes clusters on bare metal just for fun in just a few seconds I noticed something which made me stop reading this article all together.

Just deploy Rook and Ceph? ARE YOU BLEEPING KIDDING ME?!?

There's a job description called "Storage Engineer". These people know a little bit about Kubernetes, but are mostly specialized in everything Ceph. This tells you everything how hard it is to keep Ceph humming along in production. As a sidenote: if you want to make really good money there's also somebody called a "Ceph consultant" who is called in when SHTF. And if SHTF in a Ceph cluster, it really does.

And that's besides all the crap it takes to get and keep Kubernetes running smoothly: Kernel Optimization. Networking. Security. Storage integration. Observability. And the list goes on...

In other words, unless you are VERY well versed in a variety of topics ranging from server architecture to deep Linux knowledge and are knee deep in the usual day to day operations stuff already you are better off running Kubernetes in the cloud and leaving all the intricacies to the likes of Google, Microsoft and Amazon than trying to run a well designed cluster architecture yourself. It just isn't worth it.

Is this a joke? It's like the kind of article about kubernetes that we would have seen 10 years ago. Especially some of the ridiculous claims like you can run every service on your local k8s that you could run in the cloud. No, building managed service equivalents to run locally is not trivial.

I feel like articles like this need to come with a diagram like this to put it in the context of relevant tradeoffs.

                       High Scale/Revenue
                              │
                              │
      Managed Services        │      Self-hosted K8s
      (Overpaying but         │       (article is
      no choice)              │       pitched here)
                              │
  ────────────────────────────┼────────────────────────────
      Low capacity            │         High capacity
                              │
                              │
      Managed Services        │      Managed Services
      (Right choice -         │      (Wasting money on
      focus on product)       │       platform team)
                              │
                        Low Scale/Revenue

Or something like that. Maybe as a function of time as well but that might be harder to do in 2d.

Sure I can absolutely manage my own k8s, but there is no doubt it's easier for me to spin up postgres and ship faster on my own. At enterprise scale it's definitely a lot easier to do everything in k8s and be able to manage as many aspects as possible. I have experience of both.

You do not need kubernetes