I've found you can get pretty far with a couple of fixed nodes and scaling vertically before bringing in k8s these days.

Right now I'm running,

- podman, with quadlet to orchestrate both single containers and `pods` using their k8s-compatible yaml definition

- systemd for other services - you can control and harden services via systemd pretty well (see https://news.ycombinator.com/item?id=44937550 from the other day). I prefer using systemd directly for Java services over containers, seems to work better imo

- Pyinfra (https://pyinfra.com/) to manage and provision the VMs and services

- Fedora CoreOS as an immutable base OS with regular automatic updates

All seems to be working really well.