The tricky part of doing this in production is cloning sandboxes across nodes. You would have to snapshot the resident memory, file system (or a CoW layer on top of the rootfs), move the data across nodes, etc.

▲

Rygian 3 hours ago | parent | next [-]

If each node has its own warmed-up VM awaiting from startup, there's no need to clone across nodes.

▲

adammiribyan 7 hours ago | parent | prev | next [-]

Agreed, cross-node is the hard next step. For now single-node density gets you surprisingly far. 1000 concurrent sandboxes on one $50 box. When we need multi-node, userfaultfd with remote page fetch is the likely path.

	▲	shayonj 24 minutes ago \| parent [-]
		Cool project. +1 on userfaultfd for the multi-node path. Wrote about how uffd-based on-demand restore works wrt to my Cloud Hypervisor change [1] if you are curious. I think the the main things to watch are fault storms at resume (all vCPUs hitting missing pages at once) and handler throughput if you're serving pages over the network instead of local mmap. I think its less likely to happen when you fork a brand new VM vs say a VM that has been doing things for 5 mins. Also interestingly, Cloud Hypervisor couldn't use MAP_PRIVATE for this because it breaks VFIO/vhost-user bindings. Firecracker's simpler device model is nice for cases like this. [1] https://www.shayon.dev/post/2026/65/linux-page-faults-mmap-a...

▲

indigodaddy 11 hours ago | parent | prev [-]

Is this relevant?

https://codesandbox.io/blog/how-we-clone-a-running-vm-in-2-s...