Well, it might work for individual plugins, but it can't work for when you use plugins as extensions for other plugins. How would plugin A know it has to load B (which is an extension of A) as a prerequisite for [lazy] loading itself?

Basically, only plugin manager can have a sane graph of plugins dependencies and know how to load them because that graph is in the end defined by the user (for the plugin manager) and plugins themselves have no clue what user might want.