Depending on the number of plugins you have, you may not notice the difference in practice. I certainly don’t but I’m very much on the minimal side of things. This all said I’m definitely of the school of thought that plugin developers should be responsible for ensuring their plugins are lazy loaded, rather than leaving it up to the user, who is not as well placed to make the decision.

Well, it might work for individual plugins, but it can't work for when you use plugins as extensions for other plugins. How would plugin A know it has to load B (which is an extension of A) as a prerequisite for [lazy] loading itself?

Basically, only plugin manager can have a sane graph of plugins dependencies and know how to load them because that graph is in the end defined by the user (for the plugin manager) and plugins themselves have no clue what user might want.