| ▲ | vlovich123 18 hours ago | |||||||
Counter proof by existence - humans use significantly less energy and don’t do intelligence through an n^2 process. The human brains seem to operate closer to SSMs like Mamba with a recurrent architecture and attention is implemented using sparse distributed weights. So yes, it’s possible to have very high intelligence without O(n^2). Is it as good as transformers? Hard to tell right now. | ||||||||
| ▲ | machina_ex_deus an hour ago | parent | next [-] | |||||||
Humans do something even "worse" - all their "weight matrices" e.g. neurons activation patterns are constantly changing while they think. If every llm had to have it's weight matrix updated during inference, that would be orders of magnitude more costly than merely having n^2 attention. | ||||||||
| ▲ | ashkankiani 16 hours ago | parent | prev [-] | |||||||
he said specifically lossless attention. As in that algorithm. what you just said is irrelevant to the conversation because all of the current AI hype is built on the back of attention | ||||||||
| ||||||||