For my version of the AI assistant, I used a Docker container and Unix permissions:

https://github.com/skorokithakis/stavrobot

All plugins run in one Docker container, but they're isolated from each other by different *nix users, so they can't read each other's files. That's much more lightweight, and you don't have to run one container per plugin.

Crucially, plugins can't read each other's secrets or modify each other's code. I even have a plugin configuration webpage that doesn't go through an LLM, so the LLM never sees your secrets if you don't want to.