Remix.run Logo
fc417fc802 2 hours ago

> they lack an explicit architecture for the executive control of attention found in humans

Deceptive terminology strikes again! The "attention" mechanism in transformers appears (to my understanding at least) to have about as much to do with human attention as the "neurons" in a multi-layer perceptron have to do with biological neurons.

That said, the core premise of building in something that mimics executive function is an intriguing one (which I assume has been explored before but it's not something I'm familiar with).