Remix.run Logo
▲ vlovich123 18 hours ago

Counter proof by existence - humans use significantly less energy and don’t do intelligence through an n^2 process. The human brains seem to operate closer to SSMs like Mamba with a recurrent architecture and attention is implemented using sparse distributed weights. So yes, it’s possible to have very high intelligence without O(n^2). Is it as good as transformers? Hard to tell right now.

▲machina_ex_deus an hour ago | parent | next [-]

Humans do something even "worse" - all their "weight matrices" e.g. neurons activation patterns are constantly changing while they think. If every llm had to have it's weight matrix updated during inference, that would be orders of magnitude more costly than merely having n^2 attention.

▲ashkankiani 16 hours ago | parent | prev [-]

he said specifically lossless attention. As in that algorithm. what you just said is irrelevant to the conversation because all of the current AI hype is built on the back of attention

▲vlovich123 4 hours ago | parent [-]

Here’s what was literally quoted in the response:

> That compute will still be useful even if we find a more efficient architecture

Fixing N^2 attention would require a different architecture, but the person I was responding to was treating attention as if it’s the only architecture we have for intelligence when we know that not only do we have less competitive intelligence for cheaper, human intelligence doesn’t use n^2 attention. You have also fallen into the trap of misunderstanding the statement same as the person I replied to.