I'm always kind of blown away by experiences like this. Admittedly, I've been using Kubernetes since the early days and I manage an Infra team that operates a couple thousand self-managed Kubernetes clusters so... expert blindness at work. Before that I did everything from golden images to pushing changes via rsync and kicking a script to deploy.

Maybe it's because I adopted early and have grown with the technology it all just makes sense? It's not that complicated if you limit yourself to the core stuff. Maybe I need to write a book like "Kubernetes for Greybeards" or something like that.

What does fucking kill me in the Kubernetes ecosystem is the amount of add-on crap that is pitched as "necessary". Sidecars... so many sidecars. Please stop. There's way too much vendor garbage surrounding the ecosystem and dev's rarely stop to think about whether they should deploy something when it's easy as dropping in some YAML and letting the cluster magically run it.

Kubernetes starts sucking at first sight and is kind of the sign of the ailment of modern times.

Let me try to explain:

First, you encounter the biggest impedance mismatch between cloud and on prem: Kubernetes works with pods, while AWS works with instances as the unit of useful work, so they must map to each other, right?

Wrong, first each instance needs to run a Kubernetes node, which duplicates the management infrastructure hosted by AWS, and reduces the support for granularity, like if I need 4 cores for my workload, I start an instance with 4 cores, right?. Not so with k8s, you have to start up a big node, then schedule pods there.

Yay, extra complexity and overhead! And it's like when you need 3 carrots for the meal your cooking, but the store only sells it in packs of 8, you have to pay for the ones you don't need, and then figure out how to utilize the waste.

I'm not even going to talk about on-prem kubernetes, as I've never seen anyone IRL use it.

Another thing is that the need for kubernetes is manifestation of crappy modern dev culture. If you wrote your server in node, Python and Ruby, you're running a single threaded app in the era of even laptops having 32 core CPUs. And individual instances are slow, so you're even more dependent on scaling.

So, to make use of the extra CPU power, you're forced to turn to Docker/k8s and scale your infra that way, whereas if you went with something like Go, or god forbid, something as deeply uncool as ASP.NET, you could just put your 32 core server, and you get fast single threaded perf, and perfect and simple multi-threaded utilization out of the box without any of the headaches.

Also I've found stuff like rolling updates to be a gimmick.

Also a huge disclaimer, I don't think k8s is a fundamentally sucky or stupid thing, it's just I've never seen it used as a beneficial architectural pattern in practice.

I wouldn't throw "cloud shit" in the same bucket as Kubernetes. Professional cloud services are mostly great, but super expensive.

Kubernetes is interesting, because it basically takes everything you know and sort of pushes it down the stack (or up, depending on your viewpoint). To some extend I get the impression that the idea was: Wouldn't it be great if we took all the infrastructure stuff, and just merged the whole thing into one tool, which you can configure using a completely unsuitable markup language. The answer is that "No, that would in fact not be great".

For me the issue is that Kubernetes is overused. You can not really argue that it's not useful or has its place, but that place is much much small than the Internet wants us to believe. It's one of the few services where I feel like "Call us" would be an appropriate sales method.

The article is correct, you probably don't need Kubernetes. It's a amazing piece of technology, but it's not for the majority of us. It is and should be viewed as a niche product.

I have pretty much the exact same opinion, right down to “cloud shit” (I have even used that exact term at work to multiple levels of management, specifically “I’m not working on cloud shit, you know what I like working on and I’ll do pretty much anything else, but no cloud shit”). I have worked on too many projects where all you need is a 4-8 vCPU VM running nginx and some external database, but for some reasons there’s like 30 containers and 45 different repos and 10k lines of terraform, and the fastest requests take 750ms which should be nginx->SQL->return in <10ms. “But it’s autoscaling and load balanced!” That’s great, we have max 10k customers online at once and the most complex requests are “set this field to true” and “get me an indexed list.” This could be hosted on a raspberry pi with better performance.

But for some reason this is what people want to do. They would rather spend hours debugging kubernetes, terraform, and docker, and spending 5 digits on cloud every month, to serve what could literally be proxied authenticated DB lookups. We have “hack days” a few times a year, and I’m genuinely debating rewriting the entire “cloud” portion of our current product in gunicorn or something, host it on a $50/month vps, point it at a mirror of our prod db, and see how many orders of magnitude of performance I can knock off in a day.

I’ve only managed to convert one “cloud person” to my viewpoint but it was quite entertaining. I was demoing a side project[0] that involved pulling data from ~6 different sources (none hosted by me), concatenating them, deduping them, doing some math, looking up in a different source an image (unique to each displayed item), and then displaying the list of final items with images in a list or a grid. ~5k items. Load time on my fibre connection was 200-250ms, sorting/filtering was <100ms. As I was demoing this, a few people asked about the architecture, and one didn’t believe that it was a 750 line python file (using multiprocessing, admittedly) hosted on an 8 core VPS until I literally showed up. He didn’t believe it was possible to have this kind of performance in a “legacy monolithic” (his words) application.

I think it’s so heavily ingrained in most cloud/web developers that this is the only option that they will not even entertain the thought that it can be done another way.

[0] This particular project failed for other reasons, and is no longer live.

IMO the actual experience you want is something as simple as you can do with those old-school simple PHP-based "serverless" web hosters. You just upload your stuff and maybe configure a database URL and that's it.

Regarding AWS, function.zip Lambda + DynamoDB + S3 + SQS is basically this at "enterprise-grade".

Now you have the in-between left, where you want enterprise-grade (availability, scaling and fault tolerance) but with a catch (like lower costs, more control, data sovereignty or things that are not as serverless as you want, e.g. search engine etc.). In these cases you will run into many trade-off decisions, that may lead you to K8s, cloud specific stacks and also simple VM ClickOps / ShellOps setups.

As long as you can still be on the pet instead of cattle range of problems, K8s is not what you want. But as soon as you want cattle, reproducible throw-away online environments etc. "just have a VM or a physical server" will become a "build your own private cloud solution" that may or may not become a bad private cloud / K8s.

Kubernetes was developed by Google, and it seems to me it's a classic example of a technology that can work well in a resource-rich (people, money) environment. But in an averge business it becomes a time/money sink quite quickly.

Way too many businesses cargo-cult themselves into thinking that they need FAANG-class infra, even though they haven't got the same scale or the same level of resourcing. Devs and ops people love it because they get to cosplay and also get the right words on their CV.

If you're not Google-scale then, as you say, a few VMs or physical boxes are all you need for most systems. But its not sexy, so the business needs people who don't mind that.

I agree, but what pisses me the most is that today higher level abstractions (like cloud, spring boost,...) are hiding lower level functionality so well, that you are literally forced to use obnoxious amounts of time to study documentation (if you are in luck and it is well written), while everything is decorated with new naming of known concepts that was invented by people who didn't know that the concept already exists and has a name or some marketing guy figured out it will sell better with more "cool" name.

Like Shakespeare work would be clumsy and half translated to french advertising jargoon and you are forced to read it and make it work on a stage.

I think a good compromise is building your applications in a containerised manner so that you can simply run them with docker-compose, but then if it turns out you need some heavy scale, it's merely a case of switching the underlying infrastructure.

That said, my experience has been fairly different. Running microk8s on a VPS with some containers and an ingress just seems like a nice way to spin them up as pods and manage them. It really doesn't feel that complicated.

Once you integrate with cloud providers and you get more complex systems, sure, it gets more complex.

I much prefer the container paradigm for separating parts of apps out and managing dependencies and reproducibility. Having a bunch of raw servers with all the bullshit and drift and mutability that comes along with it feels far more like a headache to me than the abstraction of k8s or docker.

If you aren't deploying stuff with GitOps/IaC in 2024 I fear for anyone that has to manage or maintain it. 100% reproducibility and declarative infrastructure are wonderful things I could never come back from.

I don’t mind the cloud, but even in enterprise organisations I fail to see the value of a lot of the more complex tools. I’ve anlways worked with Azure because Denmark is basically Microsoft territory in a lot of non-tech organisations because of the synergy between pricing and IT operations staff.

I’ve done bicep, terraform and both Kubernetes and the managed (I forgot what azure conteiner apps running on top of what is basically Kubernetes is called). When I can get away with it I always use the Azure CLI through bash scripts in a pipeline however and build directly into Azure App services for contained which is just so much less complicated than what you probably call “cloud shit”. The cool part about the Azure CLI and their app services is that it hasn’t really changed in the past 3 years, and they are almost one size fit any organisation. So all anyone needs to update in the YAML scripts are the variables. By contrast working with Bicep/Terraform, Jenkins and whatever else people use has been absolutely horrible, sometimes requiring full time staff just to keep it updated. I suppose it may be better now that azure co-pilot can probably auto-generate what you need. A complete waste of resources in my opinion. It used to be more expensive, but with the last price hike of 600% on azure container apps it’s usually cheaper. It’s also way more cost efficient in terms of maintaining since it’ll just work after the initial setup pipeline has run. This is the only way I have found that is easier than what it was when organisations ran their own servers. Whether it was in the basement or at some local hardware house (not exactly sure what you call the places where you rent server rack space). Well places like Digital Ocean are even easier but they aren’t used by enterprise.

I’m fairly certain I’ve ever worked with an organisation that needed anything more than that since basically nothing in Denmark scales beyond what can run on a couple of servers behind a load balancer. One of the few exceptions is the tax system which sees almost 0 usage except for the couple of weeks where the entire adult population logs in in at the same time. When DevOps teams push back, I tend to remind them that StackOverflow ran on a couple of IIS servers for a while and that they don’t have even 10% of the users.

Eventually the business case for Azure will push people back to renting hardware space or jumping to Hetzner and similar. But that’s a different story.

I am running VPSes at our small startup-ish company on IaaS cloud.

Every time we get a new guy I have to explain that we are already „in cloud” there is no need to „move to cloud”.

Cloud is a broad spectrum of tools. I think the Goldilocks zone is something right in between bare-ass VMs and FaaS.

If I can restart the machine and it magically resolves the underlying hardware fault (presumably by migrating me to a new host), then I am in a happy place.

Most of the other problems can be dealt with using modern tooling. I lean aggressively on things like SQLite and self-contained deployments in .NET to reduce complexity at hosting time.

When you can deploy your entire solution with a single zip file you would be crazy to go for something like K8s.

One other cloud thing that is useful is S3-style services. Clever use of these APIs can provide incredible horizontal scalability for a single instance solution. SQLite on NVMe is very fast if you are offloading your blobs to another provider.

> If I get to tens of millions of users, maybe I’ll worry about it then.

Nope, then you'll set up sharded databases and a bunch of application servers behind a load balancer.

I hate "cloud shit" as well, though specifically that there's a vendor-specific 'app', or terminology, or both, for everything that we've had standard terms for, for decades.

I just want a damn network, a couple of virtual machines, and a database. Why does each <cloud provider> have to create different fancy wrappers over everything, that not even their own sales consultants, and even engineers, understand?(1)

What I do like about Docker and Kubernetes is that shifting from one cloud provider to another, or even back to on-premises (I'm waiting for the day our organisation says "<cloud-provider> is too damn expensive; those damn management consultants lied to us!!!!") is a lot easier than re-building <cloud provider>'s bespoke shit in <another cloud provider>'s bespoke shit, or back on-premises with real tech (the right option in my opinion for anyone less than a (truly) global presence).

I do like the feel of, and being able to touch bare metal, though the 180-proof-ether-based container stuff is nice for quick, flexible, and (sometimes) dirty. Especially for experimenting when the Directors for War and Finance (the same person) say "We don't have the budget!! We're not buying another server/more RAM/another disk/newer CPUs! Get fucked!".

The other thing about Docker specifically I like is I can 'shop' around for boilerplate templates that I can then customise without having to screw around manually building/installing them from scratch. And if I fuck it up, I just delete the container and spin up another one from the image.

(1) The answer is 'vendor lock-in', kids.

(I apologise, I've had a looooooong day today.......)

I am sure the consultants and others peddling Kubernetes like it, since they are getting the paycheck for bringing it on, or cementing their position, as they are the ones with the expert knowledge of the system then, if anyone is at all, after something has been migrated to Kubernetes.

All of these anecdotes seem to come from people who don’t bother to try to learn kubernetes.

> YAML files, and then spend a day fixing them by copy-pasting increasingly-convoluted things on stackexchange.

This is terrible behavior. Its not any different from yanking out pam modules because you’re getting SSH auth failures caused by a bad permission on an SSH key.

> If I get to tens of millions of users, maybe I’ll worry about it then.

K8s isn’t there for 10s of millions of users. It’s there so you’re not dependent on some bespoke VM state. It also allows you to do code review on infra changes like port numbers being exposed, etc.

Separately, your VM likely isn’t coming from any standard build pipeline so now a vulnerability patch is a login to the machine and an update, which hopefully leaves it in the same state as VMs created new…

Oh, and assuming you don’t want to take downtime on every update, you’ll want a few replicas and load balancing across them (or active/passive HA at a minimum). Good luck representing that as reviewable code as well if you are running VMs.

The people that don’t understand the value prop of infra as code orchestration systems like k8s tend to work in environments where “maintenance downtime” is acceptable and there are only one or two people that actually adjust the configurations.

I have same thoughts.

the only form of kubernetes I would be willing to try is the one with kata-containers for having all the security of virual machines.

Or if you got a static ip and fast up speed, then just port forward 80, 443 and start hosting your self. Even an old Intel Mac Book pro from 2000's with 4 GB of RAM may not be that hot running MacOS, but install Debian with no X. It is running smooth as a whistle, while running several conduit matrix, haraka, zone-mta, ice cast, nginx with no issues.

WebRTC/TUN/STUN becomes an issue with the nginx config. May consider looking at pingora. The whole rust -> binary + toml file is super nice to run from system admin perspective.

“let me install everything myself” doesn’t generalise well and gets messy even if you IaC it.

There is a reason k8s gang keeps going on about “cattle not pets”. The starting assumptions and goals are fundamentally different vs “give me a physical server”

Both have their place I think so not really one is right other is wrong

I don't mind K8s, and it can be usefull if you just stay to the core essentials. But I do agree that there seems to be the sentiment:"Everybody has to like it!"

And I mostly think that this is because our collective bias for everything Big Tech does is good. While often it just depends. Just because Google does X or Y doesn't mean it will work for everybody else.

The yaml and the k8s model can be grokked it just takes a willingness to.

That being said there are abstractions terraform, pulumi, others.

But my go to is always that most companies will never get to the point where k8s is required — most companies never get to that scale. A well maintained docker compose setup gets you a long way.

This read as "old man yells at cloud" to me.

I've managed a few thousand VMs in the past, and I'm extremely grateful for it. An image is built in CI, service declares what it needs, the scheduler just handles shit. I'm paged significantly less and things are predictable and consistent unlike the world of VMs where even your best attempt at configuration management would result in drift, because the CM system is only enforcing a subset of everything that could go wrong.

But yes, Kubernetes is configured in YAML, and YAML kind of sucks, but you rarely do that. The thing that changes is your code, and once you've got the boilerplate down CI does the rest.

I’m confused. Do you hate cloud or the container shit? Cloud is awesome. Containers is dumb as hell for 99% of the applications hacker news kids work on.

One thing that might help you in this madness is:

https://github.com/dhall-lang/dhall-kubernetes

Type safe, fat finger safe representation of your YAMLs is grossly underrated.

In my limited experience k8s is treated as some magic instead of what it is: resource virtualization platform. Sort one (or two) step above hardware virtualization platforms (e.g. esxi).

The fact that vmware can migrate a running vm to a different hardware node is surely powerful feature. Do you want to pay for that with complexity? If you are one infra team serving on-prem deployments with consistent loads you provision things when new projects are started and things break. However, if infra team serves internal product teams it is nice to give certain guarantees to them and limit blast radius how they can affect other teams.

This is where kubernetes sit. It's a deployment platform, where the deployable is container image instead of VM image. Slice off a namespace to an internal team and have their deployment blast radius contained within their namespace.

Do you need such flexibility? I'm pretty sure that roughly 99% of all teams do not. A static set of VMs provisioned with some container orchestration system is more than enough for them. Loads and deployments are too static for it to matter.

>>> But it allows seamless upgrades!

Dude or dudette, if you can't properly drain your nodes and migrate sessions now, kubernetes will not save you.

I'm sure you can get some of the handy extras that come with a typical kubernetes deployment without the kubernetes, but overall I'll take kubernetes + cloud. Once you've got the hang of it, it's ok. I have a terraform project that deploys clusters with external-dns, external-secrets, cert-manager, metrics, monitoring stack, scalers and FluxCD. From there, pretty much everything else is done via FluxCD (workloads, dashboards, alerts). And while I detest writing helm charts (and sometimes using them, as they can get "stuck" in several ways) they do allow you to wrap up a lot of the kubernetes components into a single package that accepts more-or-less standardized yaml for stuff like resource limits, annotations (eg for granting and AWS role to a service) etc. And FluxCD .postBuild is extremely handy for defining environment vars to apply to more generic template yaml, so we avoid a sprawl. So much so that I am the one-man-band (Sys|Dev|Sec)Ops for our small company, and that doesn't give me panic attacks.

The cloud integration part can be hairy but I have terraform patterns that, once worked out, are cookie cutter.

With cloud kubernetes, I can imagine starting from scratch, taking a wrong turn and ending up in hell.

But I'm exchanging one problem set for another. Having spent years managing fleets of physical and virtual servers, I'm happier and more productive now. I never need to worry about building systems or automation for doing OS build / patching, config management, application packaging and deployment, secrets management, service discovery, external DNS, load balancering, TSL certs etc. Because while those are just "words" now, back then each one was a huge project involving multiple people fighting over "CentOS Vs Ubuntu", "Puppet Vs Ansible", "RPMs Vs docker containers", "Patching Vs swapping AMIs". If you're using Consul and Vault, good luck - you have to integrate all of that into whatever mess you've built, and you'll likely have to write puppet code and scripts to hook it all up together. I lost a chunk of my life writing 'dockerctl' and a bunch of puppet code that deployed it so it could manage docker containers as systemd services. Then building a vault integration for that. It worked great across multiple data centers, but took considerable effort. And in the end it's a unique snowflake used by exactly one company, hardly documented and likely full of undiscovered bugs and race conditions even after all the hard work. The time it took to onboard new engineers was considerable and it took time away from an existing engineer. And we still had certificates expire in production.

> I might just be stupid, but it feels like all I ever do with ____ is update and break ____ files, and then spend a day fixing them by copy-pasting increasingly-convoluted things on stackexchange. I cannot imagine how anyone goes to work and actually enjoys working in ____, though I guess someone must in terms of “law of large numbers”.

I'd make a similar statement about the sys admin stuff you already know well. Give me yaml and a rest api any day.

I see where you and the article are coming from, though. The article reasonably points out that k8s is heavy for simpler workloads.

I find it always interesting and weird to read so dimetrical points to kubernetes.

1. Cloud for me is a lot better than what we had before: Before i had to create a ticket for our internal it department, have huge cross-charges (like 500$ for a server, instead of 50), had to wait for a few weeks and than get lectured that installing basic default tools on that suse based server would take a week and add additional cross-charges onto it.

Their reasoning? Oh we do backup and upgrades...

With cloud, i click a server in a Minute for less money, i upgrade it myself and have snapshots as a basic backup workflow which actually is reliable and works.

Then k8s came along and let me be clear: my k8s setup is big, so its definitly worth it but tx to my experience with a bigger k8s setup, my very small ones are also working very very well. I get, out of the box, HA, network policies, snapshotting through my selected storage setup, Infrastructure as code etc.

Instead of having shitty shell scripts, ansible setup and co, i only write a little bit of yaml, check it into my git system and roll it out with ansible. Absolut no brainer.

And the auto healing solved real issues: Out of memroy? just restart the pod. Out of disk? Just recreate it. Logging and metrics just works out of the box thanks to the prometheus based monitoring stack.

Starting with one server, yeah why not but you know you are not doing it right if you are the only one who can set it up and recover it. But if you don't have anyone with expertise, i would also not just start with k8s.

If my startup is a pure application + db thingy, i would go with any app platform out there. But we have real servers because we do stuff with data and need sometimes performance, performance is expensive if you run it in the cloud.

I'll let you in on the joke. The joke is the demand for 100% availability and instant gratification. we're making services where anything less than 4 nines, which is 5 minutes month, is deemed unacceptable. three nines is 10 minutes a week. two nines is 15 minutes a day. there are some things that are important enough that you can't take a coffee break and wait for, but Kubernetes lets you push four nines of availability, no problem. Kubernetes is solving for that level of availability, but my own body doesn't have anything near that level of availability. demanding that from everything and everyone else is what pushes for Kubernetes level of complexity.

Isn't using VMs or servers just cloud shit in a different form? How does that fix anything? Orchestrating software (or hardware) is a pain in the ass whether it's in your closet or someone else's.

Once it gets hard to run the needed services for all the applications in the organisation/team on your development machine it'll start to look more attractive to turn to Docker/Podman and the like, and that's when automatic tests and deploys built on disgusting YAML starts to make more sense.

I've been at a place where the two main applications were tightly coupled with several support systems and they all were dependent on one or more of Oracle DB, Postgres, Redis, JBoss, Tibco EMS and quite a bit more. Good luck using your development device to boot and run the test suite without containers. Before that team started putting stuff in containers they used the CI/CD environment to run the full set of tests, so they needed to do a PR, get it accepted, maybe wait for someone else's test run to finish, then watch it run, and if something blew, go back to commit, push to PR, &c. all over again.

Quite the nuisance. A full test suite had a run time of about an hour too. When I left we'd pushed it to forty minutes on our dev machines. They didn't use raw Kubernetes though, they had RedHat buy-in and used OpenShift, which is a bit more sane. But it's still a YAML nightmare that cuts you with a thousand inscrutable error messages.

Sorry to have to tell you this, but you’re old. Your neural plasticity has gone down and you feel like you have seen it al before. As a result you cling to the old and never feel like you grasp the new. The only reasonable thing to is to acknowledge and accept this and try not let it get in your way.

Our generation has seen many things before, but at the same time the world has completely changed and it’s led to the people growing up in it to be different.

You and me didn’t fully grasp CPUs anymore. Some people today don’t grasp all the details of the abstractions below K8s anymore and use it when perhaps something simpler (in architecture , not necessarily in use!) could do it better. And yet, they build wonderous things. Without editing php.ini and messing up 2 services to get one working.

Do I think K8s is the end all? Certainly not, I agree it’s sometimes overkill. But I bet you’ll like it’s follow-up tech even less. It is the way of things.

Even for on-prem scenario, I'd rather maintain a K8S control plane and let developer teams manage their own apps deployment in their own little namespace, than provisioning a bunch of new VMs each time a team need some services deployed.

For most small setups, the cost of running an empty kubernetes cluster (managed) is typically higher than setting up a db, a couple of vms and a loadbalancer, which goes a long way for running a simple service. Add some buckets, a CDN and you are pretty much good to go.

If you need dedicated people just to stay on top of running your services, you have a problem that's costing you hundreds of thousands per year. There's a lot of fun and easy stuff you can do with that kind of money. This is a pattern I see with a lot of teams that get sucked into using Kubernetes, micro services, terraform, etc. Once you need a few people just to stay on top of the complexity that comes from that, you are already spending a lot. I tend to keep things simple on my own projects because any amount of time I spend on that, I'm not spending on more valuable work like adding features, fixing bugs, etc.

Of course it's not black and white and there's always a trade off between over and under engineering. But a lot of teams default to over engineering simply by using Kubernetes from day one. You don't actually need to. There's nothing wrong with a monolith running on two simple vms with a load balancer in front of it. Worked fine twenty years ago and it is still perfectly valid. And it's dead easy to setup and manage in most popular cloud environments. If you use some kind of scaling group, it will scale just fine.

Most control planes are not free anymore, they cost like 70$/mo on AWS & GCP. Used to be a while back.

If you do find yourself wanting to create a cluster by hand, it's probably because you don't actually need lots of machines in the "cluster". In my experience it's super handy to run tests on a single-node "cluster", and then k3s is super simple. It takes something like 8 seconds to install k3s on a bare CI/CD instance, and then you can install your YAML and see that it works.

Once you're used to it, the high-level abstractions of k8 are wonderful. I run k3s on raspberry pi's because it takes care of all sorts of stuff for you, and it's easy to port code and design patterns from the big backend service to a little home project.

K8s doesn't help you solve your geographical region problem, because the geographical region problem is not running appserver instances in multiple regions. Almost any PaaS will do that for you out of the box, with way less fuss than k8s. The hard part is distributing your data.

Less overhead than writing your own glue code, less overhead than learning Kubernetes, is just use a PaaS like Google App Engine, Amazon Elastic Beanstalk, Digital Ocean App Platform, or Heroku. You have access to the same distributed databases you would with k8s.

Cloud Run is PaaS for people that like Docker. If you don't even want to climb that learning curve, try one of the others.

Yep, it's a cluster OS. If you need to run a cluster, you need to explore and understand the trade offs of k8s versus other approaches. Personally I run a small cluster on k3s, for internal tools, and love it. Yes it's a load of new abstractions to learn, but once learnt if really helps in designing large scalable systems. I manage lots of pet machines and VMs for clients, and it would be soooo much easier on k8.

I like to describe it similarly, but as a way of building platforms.

Exactly. I've come to describe k8s as a distributed operating system for servers.

K8s tries to abstract away individual "servers" and gives you an API to interact with all the compute/storage in the cluster.

> Containers are an implementation detail.

They really aren't.

Personally I have a big Nix derivation to deploy my (heterogeneous) cluster to bare metal.

None of the k8s concepts or ideas apply here.

What is the container orchestration tool of choice beyond docker swarm, then?

This has got to be the most out there k8s take I've read in a while. k8s doesn't save you from learning your cloud providers infrastructure, you have to learn k8s in addition to your cloud provider's infrastructure. It's all ALBs, ASGs, Security Groups, EBS Volumbes and IAM policy underneath and k8s, while very clever, isn't so clever as to abstract much of any of it away from you. On EKS you get to enjoy more odd limitations with your nodes than EC2 would give you on its own.

You're already building on a cluster, your cloud provider's hypervisor. They'll literally build virtual compute of any size and shape for you on demand out of heterogeneous hardware and the security guarantees are much stronger than colocated containers on k8s nodes.

There are quite a few steps between single server and k8s.

It's just the hype moving on.

Every generation has to make similar mistakes again and again.

I am sure if we had the opportunity and the hype was there we would've used k8s in 2005 as well.

The same thing is true for e.g. JavaScript on the frontend.

I am currently migrating a project from React to HTMX.

Suddenly there is no build step anymore.

Some people were like: "That's possible?"

Yes, yes it is and it turns out for that project it increases stability and makes everything less complex while adding the exact same business value.

Does that mean that React is always the wrong choice?

Well, yes, React sucks, but solutions like React? No! It depends on what you need, on the project!

Just as a carpenter doesn't use a hammer to saw, we as a profession should strive to use the right tool for the right job. (Albeit it's less clear than for the carpenter, granted)

People gravely miss-understand containerization and Docker.

All it lets you do is put shell commands into a text file and be able to run it self-contained anywhere. What is there to hate?

You still use the same local filesystem, the same host networking, still rsync your data dir, still use the same external MySQL server even if you want -- nothing has changed.

You do NOT need a load balancer, a control plane, networked storage, Kubernetes or any of that. You ADD ON those things when you want them like you add on optional heated seats to your car.

I've worked at a few tiny startups, and I've both manually administered a single server and run small k8s clusters. k8s is way easier. I think I've spent 1, maybe 2 hours on devops this year. It's not a full-time job, it's not a part-time job, it's not even an unpaid internship. Perhaps at a bigger company with more resources and odd requirements...

So, let's say you want to deploy server instances. Let's keep it simple and say you want to have 2 instances running. You want to have zero-downtime-deployment. And you want to have these 2 instances be able to access configuration (that contains secrets). You want load balancing, with the option to integrate an external load balancer. And, last, you want to be able to run this setup both locally and also on at least 2 cloud providers. (EDIT: I meant to be able to run it on 2 cloud providers. Meaning, one at a time, not both at the same time. The idea is that it's easy to migrate if necessary)

This is certainly a small subset of what kubernetes offers, but I'm curious, what would be your goto-solution for those requirements?

The attraction of this stuff is mostly the ability to keep your infrastructure configurations as code. However, I have previously checked in my systemd cofig files for projects and set up a script to pull them on new systems.

It's not clear that docker-compose or even kubernetes* is that much more complicated if you are only running 3 things.

* if you are an experienced user

Kubernetes, as an industry standard that a lot of people complain about is just a sitting duck waiting to be disrupted.

Anybody who doesn't have the money, time or engineering resources will jump on whatever appear as a decent alternative.

My intuition is that alternative already exist but I can't see it...

A bit like Spring emerged as an alternative to J2EE or what HTMX is to React & co.

Is it k3s or something more radical?

Is it on a chinese Github?

As an industry, we spent so much time sharpening our saw that we nearly forgot to cut down the tree.

ZIRP is over.

Start-ups that don't need to scale will quickly go away, because how else are you going to make a profit?

How have you been going since 2005 and still not understand the economics of software?

Cloud Run has had network egress control for a while: https://cloud.google.com/run/docs/configuring/vpc-direct-vpc

kubernetes is how I keep compute costs in 2-3 digits :V

I’m surprised Cloud Run doesn’t let you do this. You can put an AWS lambda in a VPC no problem.

> an out of the box Postgres database can't just handle unlimited connections and so connecting to it from GCR without having a DB connection proxy like PG bouncer risks exhausting the connection pool.

Good point. How many connections can it handle? Seems like it's up to 262142 in theory? Or am I reading this wrong: https://cloud.google.com/sql/docs/postgres/flags#postgres-m ??

But even 1000 seems ok? 1 per container, so 1000 running containers? Quite a lot in my world, especially since they can be quite beefy. Would be very worried about the cost way before 1000 simultaneously running containers :)

Why are GCR and pgbouncer incompatible? Could you run a pgbouncer instance in GCR?

lol, even if you have complex apps there are always easier solutions than Kubernetes. It is evident that you have never run such an app and are just talking about it. Otherwise, you would know the issues you would encounter with every update due to breaking changes. Not to mention that you need a high level of expertise and a dedicated team, which costs far more than running an app on Fargate. Recommending a managed Kubernetes cluster is nonsense, as it goes against the whole purpose of Kubernetes itself.

Kubernetes has a proportional learning curve.

If you're used to managing platforms e.g. networking, load balancers, security etc. then it's intuitive and easy.

If you're used to everything being managed for you then it will feel steep.

Using it to run easily-scalable Spark clusters. Previously used it for large distributed builds. It's been pretty great (even if annoying at times).

And there's very few investment points below it.

You can cobble together your own unique special combination of services to run apps on! It's an open ended adventure into itself!

I'm all for folks doing less, if it makes sense! But there's basically nothing except slapping together the bits yourself & convincing yourself your unique home-made system is fine. You'll be going it alone, & figuring out on the fly, all to save yourself from getting good at the one effort that has a broad community, practitioners, and massive extensibility via CRD & operators.

As I hope decent system administrator... turns out I find running minimal kubernetes setup (k3s is great there) more sensible and cheaper than managing the server in classic way.

I agree with you, but can offer a counterpoint. A system administrator might setup a server in a certain way, and if they leave the replacement has a hard time discovering what they did.

With IaC, all decisions are templated in the code and the replacement has full insight into the state of the machine.

> The push subscription will just ramp up attempting to DoS your Cloud Run service

Interesting. My assumption would be that Cloud Run should quickly* spin up more containers to handle the spike and then spin them down again. So there would be no need for back pressure? Guess it depends on the scale? How big of a spike are we talking about? :)

*Let's say a few seconds

I don't think it makes sense to conflate vendor lock-in with taking a dependency on a given technology. Do we then have "Linux lock-in" and "Postgres lock-in"? The term "lock-in" shouldn't be stretched to cover this concept imo.

Doing the same, grabbed a reasonably cheap Ryzen (zen2) server with 64GB ECC and 4x NVMe SSDs (2x 512G + 2x 1024G).

Runs pretty much this stack:

  "Infrastructure":

  - NixOS with ZFS-on-Linux for as 2 mirrors on the NVMes 
  - k3s (k8s 1.31)
  - openebs-zfs provisioner (2 storage classes, one normal and one optimized for postgres)
  - cnpg (cloud native postgres) operator for handling databases
  - k3s' built-in traefik for ingress
  - tailscale operator for remote access to cluster control plane and traefik dashboard
  - External DNS controler to automate DNS
  - Certmanager to handle LetsEncrypt
  - Grafana cloud stack for monitoring. (metrics, logs, tracing)

  Deployed stuff:
  - Essentially 4 tenants right now
  - 2x Keycloak + Postgres (2 diff. tenants)
  - 2x headscale instances with postgres (2 diff. tenants, connected to keycloak for SSO)
  - 1 Gitea with Postgres and memcached (for 1 tenant)
  - 3 postfix instances providing simple email forwarding to sendgrid (3 diff. tenants)
  - 2x dashy as homepage behind SSO for end users (2 tenants)
  - 1x Zitadel with Postgres (1 tenant, going to migrate keycloaks to it as shared service)
  - Youtrack server (1 tenant)
  - Nextcloud with postgres and redis (1 tenant)
  - tailscale-based proxy to bridge gitea and some machines that have issues getting through broken networks

The server is barely loaded and I can easily clone services around (in fact a lot of the services above? instantiated from jsonnet templates).

Deploying some stuff was more annoying than doing it by hand from shell (specifically nextcloud) but now I have replicable setup, for example if I decide to move from host to host.

Biggest downtime ever was dealing with not well documented systemd-boot behaviour which caused the server to revert to older configuration and not apply newer ones.

Do you have a write-up about this that you have to share, even if it's someone else's? I'd be curious to try this out.

Nice, that is my current approach on my latest project (referring to what your linked repo does). Inlining it here for others: copy a .env and a docker-compose.yml, run `docker compose "$@"` on the instance via ssh. That lets me deploy, run one-off commands, and tail logs easily.

People don’t want solutions, they want holistic experiences. Kubernetes feels bad and a pile of scripts feels good. Proudly declaring how much you buy into feelings feels even better!

I work with Kubernetes enough that I would answer to the title "kubernetes developer" and I would recommend you don't use kubernetes. In the same way I would recommend you don't use a car when you can walk.

Your friend lives 1/8 miles away. You go to see them every day so why wouldn't you drive? Well, cars are expensive and you should avoid them if you don't need them. There are a TON of downsides to driving a car 1/4 of a mile every day. And there are a TON of benefits to using a car to drive 25 miles every day.

I hate to quash a good debate but this all falls under the predictable but often forgotten "it depends". Usually do you need kubernetes == do you have a lot of shit to run.

TL;DR someone didn't learn to plan capacity.

Nope.

Busy using a fair bit of A.I. to produce yams manifests. It still hallucinates a fair bit and doesn’t connect the dots well between the various parts of you colour outside the lines even slightly

Works but feels substantially weaker than on coding tasks. Not sure whether that’s lack of training material or the lack of execution flow code has.

Hard no.

There is already people who uses k8s that does not understand how it works or even know how the app works inside k8s.

Adding AI to it is just asking for a disaster down the line.

That probably runs on Kubernetes (or Borg) under the hood.

1. Do some capacity planning

2. Apply appropriate changes to application resources (like parameters for spreading pods around)

3. Add descheduler[1] or similar tool to force redistribution of pods

4. Configure your cluster autoscaling params according to values from step (1) and have it autoscale before nodes are too heavily loaded.

Dokku runs on multiple servers by abstracting other schedulers (kubernetes, nomad). You don't really need to deal with the scheduler side (other than setting it up on the new servers in your cluster, which is a single command we document).

Disclaimer: I am the Dokku maintainer

Well, since it's cloud run...

... you spin knative on kubernetes cluster and copy over the actually-k8s-manifests over :P

The "...You probably don't either" part is where the argument loses all of its weight. How do they know what I or anyone else needs?

I've never heard of this. Can you share a link of what you are referring to? I care about this.

Yes, but it's based on KNative (actually it's a managed KNative service[1]) which is based on K8s. You can get a dev experience similar to CloudRun with KNative on K8s.

[1]: https://cloud.google.com/blog/products/serverless/knative-ba...