"a single, larger layer of hidden units"

Does this not mean that the entire model must cycle to operate any given part? Division into concurrent "modules" (the term appearing in this paper,) affords optimizing frequency independently and intentionally.

Also, what certainty is there that everything is best modelled with multilayer NN? Diversity of algorithms, independently optimized, could yield benefits.

Further, can we hope that modularity will create useful points of observability? The inherent serialization that develops between modules could be analyzed, and possibly reveal great insights.

Finally, isn't there a possibility that AGI could be achieved more rapidly by factoring the various processes into discrete modules, as opposed to solving every conceivable difficulty in a monolithic manner, whatever the algorithm?

That's a lot of questions. Seems like identifying possible benefits is easy enough that this approach is worthwhile exploring. We shall see I suppose. At the very least we know the modularization of HRM has a valid precedent: real biological brains.

It would not surprise me if all of these tangential advances in various models and approaches ultimately become part of a larger frame work of modules designed to handle certain tasks - similar to how your medula oblongata operates breathing and heart rate, and your amygdala sorts out memory and hormone production, and your cingulate gyrus helps control motor function, et al.

We have a great example (us), we just need to hone and replicate it.