I feel like I'm missing something here when this is being used with nomad. Caveat being that the only comparable technologies I've worked with are k8s and ECS. In the article they mention that they are using a containerd shim to launch micro VMs, so from the perspective of the scheduler, whether the VM is actually "sleeping" or not, it looks like it's running since they continue to respond to health checks. So what exactly is the point of suspending the VMs on idle if the scheduler still thinks they're running? Whatever memory is reserved for that job is still going to be reserved, so you're not able to oversubscribe the host regardless.

▲

nicoche 5 hours ago | parent [-]

Hey!

You got everything correctly. The advantages are: - For the end-user: not paying or paying less - For the hypervisor owner: a sleeping instance uses no CPU, so it reduces the load on the hypervisor

Other than that, it's still possible to oversubscribe, but you're right, we need to trump the scheduler. Another cool thing is that in the worst case scenario where an hypervisor gets full and it's over capacity, sleeping instances are great candidates for eviction.

▲

nevon 5 hours ago | parent [-]

Ah, I think the part that I didn't consider was that an "idle" VM is not zero CPU cost, unlike a container, so indeed from a hypervisor owner perspective you'd like other active VMs to be able to use that CPU time. But again, doesn't that presuppose oversubscription? If a node is fully reserved, it doesn't matter if all of the running VMs are idle, you're still not going to be able to schedule another job on that node, so your costs remain the same unless you oversubscribe the host and count on the fact that there will be unused capacity available most of the time (similar to AWS Flex instances).

	▲	nicoche 5 hours ago \| parent [-]
		Yes definitely as an operator, you want to oversubscribe hosts. What I was mentioning is that there are still small benefits when an host is not full: the CPU gains _and_, for users, the fact that they're not paying/paying less (even though the operator is still paying for the full underutilized hypervisor, but hey, that's the game)