I don't think a modern LLM is necessarily less complicated than a squirrel brain. If anything it's more engineered (well structured and dissectable), but loaded with tons of erroneous circuitry that is completely irrelevant for intelligence.

The squirrel brain is an analogue mostly hardcoded circuit. It can take about one synapse to represent each "weight". A synapse is just a bit of fat membrane with some ion channels stuck on the surface.

A flip flop to represent a bit takes about 6 transistors, but in a typical modern GPU is going to need way more transitors to wire that bit - at least 20-30. multiply that by the minimum amount of bits to represent a single NN weight and you're looking at about 200-300 transitors just to represent one NN param for computing

And that's for actual compute. The actual weights in a GPU are stored most of the time in DRAM which needs to be constantly shuttled back and forth between the GPU's SRAM and HBM DRAM.

300 transistors with memory shuttling overhead versus a bit of fat membrane, and it's obvious general purpose GPU compute has a huge energy and compute overhead.

In the future, all 300 could conceivably replaced with a single crossbar latch in the form of a memristor.