I made this decision at a startup (albeit when the eng team was ~30 people, and we had a monolith with ~10 supporting services). I wouldn’t do it again, even for the reasons stated in the article.

The uniformity is nice, we were moving from apps running directly ec2 instances provisioned with ansible. Each time we spun up a new service it was a process to get the ec2 instances provisioned just so.

But k8s is such a pain in the ass. One thing that I think people new to it don’t realize is that it’s not at all batteries included - to get a basic managed cluster setup, you’re still going to be installing a bunch of additional controllers (ingress, cert-manager, external dns to start). And then you’re on the hook for making sure all those processes stay up (hope the admission webhook controller for a critical resource doesn’t go down!). Then you’ve got to do a major upgrade on not only your cluster, but all of those controllers every ~3 months. And no one is shy about introducing breaking changes.

Also you’re introducing a huge amount of complexity with the k8s networking and dns layer that most startups have zero need for (if you’re on EKS, make sure to read about scaling and monitoring CoreDNS).

I think there is a real hole in the market for a simple solution that lets you deploy some containers to some instances in a declarative fashion without all of that complexity and does decent LTS versions. I imagine there’s something out there that does this, but k8s has really sucked up all the oxygen.

Pretty sure if there was a simple alternative, people would hate it.

Everyone initially wants thing A. But then they want to customize it to do all permutations and combinations n of A, B, C. They want it to be extensible. They want redundancy. They want orchestration. They want integration.

It’s why practically every config file format eventually becomes its own scripting language. Even HTML started off simple — now ridiculously complex — all the more ironic since practically nobody writes it by hand. Instead of CSS simplifying it, it became more complex.

There is another thing that is extremely customizable and extensible. It’s called a programming language. People write programs to solve specific problems.

There seems to be a perverse trend of cobbling together a Byzantine mesh of libraries, plugins, and services with complex configuration files to make it do practically everything possible. We just used to write software for such purposes…

And for anyone who thinks HTML is simple… the A (anchor) tag has an “ping” attribute that results in POST requests to a list of URLs when a link is clicked ! The list of attributes and resulting variations in behavior is quite mind boggling. It was supposed to be a damn link! https://html.spec.whatwg.org/multipage/links.html

> One thing that I think people new to it don’t realize is that it’s not at all batteries included - to get a basic managed cluster setup, you’re still going to be installing a bunch of additional controllers (ingress, cert-manager, external dns to start).

And if you can do this again, what's your solution to reverse proxy, certificate management, DNS...etc? I guess you can docker-compose some custom stack on a single machine, maybe add one more machine then you can say it's HA enough for small scale. But you can also spend the same amount of time to install those kubernetes controllers with zero customization. In my experience, if you go with the default configuration, most of the well-maintained k8s components are boring as hell these days.

> (if you’re on EKS, make sure to read about scaling and monitoring CoreDNS)

If load to your service increases, you need to scale up/out your service. This is universally true. Do you have a proprietary solution that's easier and more reliable than bumping up the replicas count in kubernetes?

There are lots of design decisions in Kubernetes that I hate. But if you want me to choose between Kubernetes and any proprietary stack, in 2026, I would definitely choose Kubernetes.

Dear friend, you have built a Kubernetes

https://www.macchaffee.com/blog/2024/you-have-built-a-kubern...

> I think there is a real hole in the market for a simple solution that lets you deploy some containers to some instances in a declarative fashion without all of that complexity and does decent LTS versions

Hashicorp's Nomad basically is just that, supports various way of running stuff too which is neat. Shame about the license change which basically killed all my interest in it, so seems the hole is indeed still unfilled.

I don't know... running a startup sized kubernetes is relatively easy and pain free these days (k3s). Especially when it comes to scaling up.

CNPG is an absolute monster (in a good way). cert-manager is easier than the docker alternative, calico has never failed me (except in bgp mode which has some footguns like not being able to come back from a dead state since it has a chicken and an egg problem unless you point it to the external load balancer which I would have known if I read the documentation). trafeik is all you need. talosos largely mitigates the bare metal problems and comes pre-hardened and pre-optimized.

I solo most of my development projects and have used k3s for all of them. The only complaint is that cert-manager by default will fail silently and your certificates will expire. I largely mitigated this by having proper visibility setup via grafana and automated alerts (warns if certificates are about to expire) which should have been done by me anyway.

Two years ago I'd agree, today with LLMs everything I have runs talos with fully automated updates and I haven't had to be on-call for almost a year.

k8s is not a pain, I would never return to something like Pupet / Ansible / etc ... to deploy bare ec2 instances, it's just re-inventing the wheel badly.

Just use ECS / Fargate with an ALB in front if you need a simpler use case.

We started out core product on ECS, which is a declarative way to run a containerized service. It has been nice and reliable, but it has limitations (slow scaling, weird AWS Quotas if you have ephemeral tasks).

We're moving our non-critical components onto EKS (pipelines, tooling, etc). We had one outage from runaway IP allocation in a subnet, but otherwise it's been pretty stable.

I do hear vague horror stories so I'm really not excited about moving our prod stack to it, but it's actually been really good for installing 3rd party software so far.

> I think there is a real hole in the market for a simple solution

Unless of course, all of the busywork that comes with kubernetes IS the value (to the engineer). Perhaps a bunch of engineers know at some level that locking the company into an overcomplicated cloud-within-a-cloud setup that has all sorts of weekly issues and requires constant work gives them a lot of job safety that they wouldn't get if they just used an AWS autoscaling group and you're done for the next 5 years.

Because simpler solutions DO exist (like a loadbalancer in front of an autoscale group, and not making a giant SOA for an app that orders you taxis, or books you a bnb or whatever nonsense).

Kamal is somewhere in the middle. Probably a little closer to a bunch of bash scripts. But it’ll get your container going pretty quick. Can take a bit of fiddling with SSH/docker-login. Plus it handles deployments very well.

I built canine.sh for exactly that reason, gives you a sensible deployment platform on top of k8s with one install, and you can customize it once you outgrow it.

> I think there is a real hole in the market for a simple solution that lets you deploy some containers to some instances in a declarative fashion without all of that complexity and does decent LTS versions. I imagine there’s something out there that does this, but k8s has really sucked up all the oxygen.

I mean, it's CDK and whatever equivalents other providers have, isn't it? If you fully embrace all the stuff they give you then it's straightforward to declare everything and it all works together. The downside is the vendor lock-in but unless you actively deploy to multiple environments, which most people don't, you're probably locked in in various ways without knowing about it.

Isn't fargate or ecs that simple service?

to what extent would AWS EKS auto mode solve those problems?

Nomad, Consul and Vault all running on VMs that you manage with Terraform.

The problem is that when you run this long enough you want K8s features anyway.