I also work in the space.

I think you’d be surprised if you look at the younger (post-2000-ish) companies. Specifically about this part:

> If you run simple workloads, it’s probably not necessary but then you’ll probably be paid for that.

You would be surprised how many workloads either are simple, or can be ham-fistedly shoved into a “simple” architecture.

Eg I’ve seen a lot of Postgres-on-k8s solutions that are basically just Postgres on bare metal with a bunch of useless abstractions. It writes to local disk, moving the pod to a new host is basically restoring from backup. Wrap it in some tool that ingests YAML and call it a day.

k8s is sort of the king of this. I hear people say they understand k8s and I can’t help but furrow my eyebrows. I don’t even pretend to, there are so many insanely complicated edge cases.

Eg if I’m sending UDP packets to a service and scale the deployment from 2 pods to 4 and the host for one of the new pods crashes and comes back up with a bad iptables rule, what are the range of potential behaviors k8s would show? What’s the difference between blocking say all traffic vs only traffic to one or two of the etcd instances? What if it’s blocking half the hosts in the cluster?

Storage has its own weird vagaries around single vs multi mounts and race conditions therein.

k8s is the king of “I don’t have to know that, the platform will do it”. I’m not sure I’ve ever heard anyone outside networking talk about whether a pod will saturate a NIC. Just chuck it in the cloud and let it figure itself out.