For the different workloads it is more that all the nodes in the cluster are the same and mixing memory with cpu intensive or io intensive workloads is hard to schedule or to get to a proper utilisation rate. Next to that when indeed setting request and limit properly it means that our Java apps use/reserve multiple cores even when handling little traffic and basically idling. Golang apps scale better there, especially towards 0.

When running on “bare” VMs each VM is its own member in the network. The pods in the cluster use an overlay network and egress is limited to egress nodes which are now shared by all workloads.

Having dedicated K8s clusters would reduce the sharing of network ingress and egress as well as choose the vm size for my workloads.