What are the reasons behind the fact that almost all of these plugin systems are so poorly engineered? Is it too much work (ie, there are no good plugin development frameworks that already enable proper isolation/permission capabilities) or "simply" a widespread lack of knowledge of what is needed, so devs learn only after their own system has been abused? Both? Something else?

Web stack plus lack of resources to architect the proper interfaces is my guess. These are software written in high level js frameworks, thus using poor dataflow patterns by default, mostly just following what is actually possible instead of employing intentional design, which would require going down some levels of abstraction and maintaining a custom fork of said frameworks. So they probably just architect plug-ins like you would instantiate a library passing a subset of the context the app uses. Basically the simplest workable thing possible. Although the disclosed hack does not mention any particular “vulnerability”. Plug-ins in obsidian are always in god mode, and the alleged hackers just tricked people in using them. Funny how an RCE waiting to happen behind a few popups is ultimately blamed on users. Shame on the developers.

You’ll need to define the security framework and building blocks that all plugins may need, which takes time to design, implement, verify and maintain.

Much easier to just skip that part.

So yes, it’s too much work (in the sense that you need to have a security-focused leadership that understands that this is a lot of work but the right thing to do).

Why does a plugin system immediately imply sandboxing?

even chrome browser plugins have security issues similar to this case. there are billions of dollars and many smart developers working on it. It's similar to building an app store inside your app. For the Apple app store, they reduce malicious apps by being very strict who/what people can publish and it's behind a paywall.