we run ~10k agent pods on k3s and went with gvisor over microvms purely for density. the memory overhead of a dedicated kernel per tenant just doesn't scale when you're trying to pack thousands of instances onto a few nodes. strict network policies and pid limits cover most of the isolation gaps anyway.

▲

alexzenla an hour ago | parent | next [-]

This is a big reason for our strategy at Edera (https://edera.dev) of building hypervisor technology that eliminates the standard x86/ARM kernel overhead in favor of deep para-virtualization.

The performance of gVisor is often a big limiting factor in deployment.

	▲	souvik1997 10 minutes ago \| parent [-]
		Edera looks very cool! Awesome team too. I read the thesis on arxiv. Do you see any limitations from using Xen instead of KVM? I think that was the biggest surprise for me as I have very rarely seen teams build on Xen.

▲

souvik1997 an hour ago | parent | prev | next [-]

Hey @clawsyndicate I'd love to learn more about your use case. We are working on a product that would potentially get you the best of both worlds (microVM security and containers/gVisor scalability). My email is in my profile.

	▲	alexzenla 44 minutes ago \| parent [-]
		This is the thesis of our research paper here, a good middle ground is necessary: https://arxiv.org/abs/2501.04580

▲

secure 3 days ago | parent | prev | next [-]

Yeah, when you run ≈10k agents instead of ≈10, you need a different solution :)

I’m curious what gVisor is getting you in your setup — of course gVisor is good for running untrusted code, but would you say that gVisor prevents issues that would otherwise make the agent break out of the kubernetes pod? Like, do you have examples you’ve observed where gVisor has saved the day?

▲

zeroxfe 4 hours ago | parent | next [-]

I've used both gVisor and microvms for this (at very large scales), and there are various tradeoffs between the two.

The huge gVisor drawback is that it __drastically_ slows down applications (despite startup time being faster.)

For agents, the startup time latency is less of an issue than the runtime cost, so microvms perform a lot better. If you're doing this in kube, then there's a bunch of other challenges to deal with if you want standard k8s features, but if you're just looking for isolated sandboxes for agents, microvms work really well.

▲

clawsyndicate 3 days ago | parent | prev [-]

since we allow agents to execute arbitrary python, we treat every container as hostile. we've definitely seen logs of agents trying to crawl /proc or hit the k8s metadata api. gvisor intercepts those syscalls so they never actually reach the host kernel.

▲

alexzenla an hour ago | parent | next [-]

The reason why virtualization approaches with true Linux kernels is still important is what you do allow via syscalls ultimately does result in a syscall on the host system, even if through layers of indirection. Ultimately, if you fork() in gVisor, that calls fork() on the host (btw fork() execve() is expensive on gVisor still).

The middle ground we've built is that a real Linux kernel interfaces with your application in the VM (we call it a zone), but that kernel then can make specialized and specific interface calls to the host system.

For example with NVIDIA on gVisor, the ioctl()'s are passed through directly, with NVIDIA driver vulnerabilities that can cause memory corruption, it leads directly into corruption in the host kernel. With our platform at Edera (https://edera.dev), the NVIDIA driver runs in the VM itself, so a memory corruption bug doesn't percolate to other systems.

▲

rootnod3 3 hours ago | parent | prev [-]

And you see no problem in that at all? Just “throw a box around it and let the potentially malicious code run”?

Wait until they find a hole. Then good luck.

	▲	alexzenla 41 minutes ago \| parent [-]
		This is why you can't build these microVM systems to just do isolation, it has to provide more value than that. Observability, policy, etc.

▲

dist-epoch 3 hours ago | parent | prev [-]

LXC containers inside a VM scales. bonus point that LXC containers feel like a VM.