> But if you actually have made things complicated enough [...]

The only problem I see in this case is that complexity doesn't come all at once. By the time you reach a problem that k8s is good at solving, you've probably already accidentally made a k8s alongside your piece of software.

In my(quite short) SWE career, I've seen software evolve, even ones with a proper design stage. Maybe I just don't have enough experience to have seen a properly designed project, but I don't know what I don't know after all.

This, and because of that, claiming your app "runs in kubernetes" is completely meaningless.

Concretely: Take your app. With one button click, or apt-get install ??? on all your machines, configure k8s. Now, run your app.

The idea that this could work has been laughable for any k8s production environment I've seen, which means you can't do things like write automated tests that inject failures into the etcd control plane, etc.

(Yes, I know there are chaos-monkey things, but they can't simulate realistic failures like kernel panics or machine reboots, because that'd impact other tenants of the Kubernetes cluster, which, realistically, is probably single tenant, but I digress..)

If your configuration is megabytes of impossible to understand YAML, and is also not portable to other environments, then what's the point?

(I understand the point for vendors in the ecosystem: People pay them for things like CNI and CSI, which replace Linux's network + storage primitives with slower, more complicated stuff that has worse fault tolerance semantics. Again, I digress...)

There are differences across vendors, but there’s a way to build with k8s where the benefit far outweighs the cost.

We run a bunch of services in two very different cloud vendors (one of which used to be DIYed with kubeadm), and also on dev machines with k3s. Takes a while to figure this out and to draw the kustomize boundaries in the right place, but once you do, it’s actually really nice.

Two things work in our favor:

- we’ve been at this for around 8 years, so we didn’t have to deal with all the gotchas at once

- we aggressively avoid tech that isn’t universal (so S3 is OK, but SQS or DynamoDB is not; use haproxy instead of ingress controllers; etc)

> Kubernetes is not just kubernetes when moving between clouds, it becomes a very opinionated product (for better or worse) with lots of vendor addons.

I think this is gradually getting better. Networking with Gateways is better than with Ingress in this sense. Things like autoscaling groups need to get better, as they are (or were a couple of years ago) very bespoke.

But k8s does not do almost any of those things!

You have to pick and then configure those components, just like you would have had to pick and configure apps doing those things if you were not using k8s, so the only thing k8s actually brings to the table is a common configuration format (yaml).

>And portable.

This made me audibly guffaw. Kubernetes is a lot of things, but "portable" is not one of them. GKE, EKS, AKS, OCP, etc., portability between them is nowhere near guaranteed.

> And there's zero setup. Just a deployment yaml that specifies exactly what you want deployed

Writing this yaml is hours and hours of setup if you can't ctrl+c/v from your last project

> Suddenly you have to become Sysadmin/SRE

I don't you made that argument but could a valid conclusion of your comment be that, because Kubernetes is so ubiquitous, using it frees you from being a Sysadmin/SRE?

> nodes running VMs,

huh, but how would bursting work then? Do VMs support it nowadays?

Kubernetes is software. It cannot do anything "itself" let alone "replace itself". Don't anthropomorphize software

Inevitably it will be a human replacing it with what they define is the best method

Well... we have k8s for that... I do not wish to take k8s away from those who like it, I am asking for a new solution that's very opinionated, and as close to zero config as practical.

I have limited ram and want scale to zero for apps that use a lot of ram, but I only use one of at a time like game servers, or things that can be done over night while I sleep like media encoding.

The main reason I went to k8s, is for the not having to think about what machine will have enough resources to run an app, just throw it at the cluster and it figures out where there's capacity. And, I want hardware failing/getting replaced to be a non issue.

edit: I wanted to add that my hobby is not systems admin, I want it to be as hands off as possible. Self-hosting is a means to an end. I have so far saved over $200/month in subscriptions by replacing subscriptions I was using with self-hosted alternatives. I can now use that money on my actual hobbies.

I am not the person you asked this question to, but I would probably do the same so I will answer:

Once you get used to it, it just makes managing things simple if you always use it for everything. I have a personal harbor service that I run on my local cluster that has all my helm charts and images, and i can run a single script that sets up my one node cluster, then run a helm install that installs cert-manager and my external-dns, and now I can deploy my app with whatever subdomain I want and I immediately get DNS set up and certs automatically provisioned and rotated. It will just work.

1. Assuming managed service, it frees me from host OS management. So basically the same proposition, as good old "PHP+MySQL" hosters. You upload your website, they make sure it works. But without limitations and with much better independence.

2. It allows me to configure everything using standard manifests. I need to provision the cluster itself initially, then everything could be done with gitops of various automation levels. I don't need to upload my pages via FTP. My CI will build OCI image, publish it to some registry, then I'll change image tag of my deployment and it'll be updated.

3. It allows to start simple, and extend seamlessly in the future. I can add new services. I can add new servers. I can add new replicas of existing services. I can add centralized logging, metrics, alerts. It'll get more complicated but I can manage the complexity and stop where I feel comfotable.

4. One big thing that's solved even with the simplest Kubernetes deployment is new version deployment with zero downtime. When I'll update image tag of my deployment, by default kubernetes will start new pod, will wait for it to answer to liveness checks, then redirect traffic to new pod, let old pod to gracefully stop and then remove it. With every alternative technology, configuring the same requires quite a bit of friction. Which naturally restricts you to deploy new versions only at blessed times. With Kubernetes, I started to trust it enough, I don't care about deployment time, I can deploy new version of heavily loaded service in the middle of the day and nobody notices.

5. There are various "add-ons" to Kubernetes which solve typical issues. For example Ingress Controller allows the developer to describe Ingress of the application. It's a set of declarative HTTP routes which will be visible outside and which will be reverse-proxied to the service inside. Simplest route is https://www.example.com/ -> http://exampleservice:8080, but there's a lot more to it, basically you can think about it as nginx config done differently. Another example is certificate manager, you install it, you configure it once to work with letsencrypt and you forget about TLS, it just works. Another example is various database controllers, for example cloudnativepg allows you to declaratively describe postgres. Controller will create pod for database, will initialize it, will create second pod, will configure it as replica, will perform continuous backup to S3, will monitor its availability and switch master to replica if necessary, will handle database upgrades. A lot of moving parts (which might be scary, tbh), all driven by a simple declarative configuration. Another example is monitoring solutions, which allow to install prometheus instance and configure it to capture all metrics from everything in cluster along with some useful charts in grafana, all with very little configuration.

6. There are various "packages" for Kubernetes which essentially package some useful software, usually in a helm charts. You can think about `apt-get` but for a more complicated set of services, mostly pre-configured and typically useful for web applications. The examples above are all installable with helm, but they add new kubernetes manifest types, which is why I called them "add-ons", but there are also simpler applications.

Just for the record, I don't suggest that to everyone. I spent quite a bit of time tinkering with Kubernetes. It definitely brings a lot of gotchas for a new user and it also requires quite a bit of self-restrictions for experienced users to not implement every devops good practice in the world. Sometimes maybe you don't even want to start with ingress, I saw cluster which used manually configured nginx reverse proxy instead and it worked for them. You can be very simple with Kubernetes.

Most of the time you don't need redundancy. You need regular backups for exceptional circumstances. And k8s gives you more complexity, and more problems through more moving parts, to give you the possibility of using a feature you'll never need, and if you do start to use it it'll probably be instead of fixing performance problems downstream

A while back when the agents got hyped I was looking into the whole "give it a VM / docker container" I realized the safest and simplest option was just to give it its own machine.

Then I realized giving it root on a $3 VPS is functionally equivalent. If it blows it up, you just reset the VM.

It sounds bad but I can't see an actual difference.