In my limited experience k8s is treated as some magic instead of what it is: resource virtualization platform. Sort one (or two) step above hardware virtualization platforms (e.g. esxi).

The fact that vmware can migrate a running vm to a different hardware node is surely powerful feature. Do you want to pay for that with complexity? If you are one infra team serving on-prem deployments with consistent loads you provision things when new projects are started and things break. However, if infra team serves internal product teams it is nice to give certain guarantees to them and limit blast radius how they can affect other teams.

This is where kubernetes sit. It's a deployment platform, where the deployable is container image instead of VM image. Slice off a namespace to an internal team and have their deployment blast radius contained within their namespace.

Do you need such flexibility? I'm pretty sure that roughly 99% of all teams do not. A static set of VMs provisioned with some container orchestration system is more than enough for them. Loads and deployments are too static for it to matter.

>>> But it allows seamless upgrades!

Dude or dudette, if you can't properly drain your nodes and migrate sessions now, kubernetes will not save you.