Remix.run Logo
▲ nightski 19 hours ago

Not only that we are one model innovation away from them being worthless. You need $1T in compute because transformers are so ungodly inefficient. That compute will still be useful even if we find a more efficient architecture, but the margin on it will be so much less.

Deepseek was a sign of this on a smaller scale.

▲xmcp123 19 hours ago | parent | next [-]

Can you expand on that deepseek part?

▲zrn900 9 hours ago | parent [-]

Deepseek concentrated on optimizing inference and it cuts up to 90%. Then published papers on how to do it. Everyone adopted it. It came as a shock to the AI bubble in the US because they bet everything on high need for compute.

Deepseek kept optimizing. Their latest paper said that by treating text as images, even more optimization could be possible as LLMs process them faster. I don't know whether they incorporated it yet.

So the parent commenter is correct: The AI bubble bet everything on the immense need for compute. It not only turned to be unprofitable, but it was also obsoleted within 1.5 years.

▲logicchains 18 hours ago | parent | prev [-]

>That compute will still be useful even if we find a more efficient architecture

Mathematically there's no way to have lossless attention without O(n^2) cost; some magical massively more efficient architecture is not going to pop up the same way nobody's going to find a way to sort an arbitrary list in log(n) time. There may still be more opportunities for incremental gains (like using the bare-minimum number of full-quadratic-attention layers necessary), but they're not going to be orders of magnitude more efficient without sacrificing intelligence.

▲vlovich123 18 hours ago | parent [-]

Counter proof by existence - humans use significantly less energy and don’t do intelligence through an n^2 process. The human brains seem to operate closer to SSMs like Mamba with a recurrent architecture and attention is implemented using sparse distributed weights. So yes, it’s possible to have very high intelligence without O(n^2). Is it as good as transformers? Hard to tell right now.

▲machina_ex_deus an hour ago | parent | next [-]

Humans do something even "worse" - all their "weight matrices" e.g. neurons activation patterns are constantly changing while they think. If every llm had to have it's weight matrix updated during inference, that would be orders of magnitude more costly than merely having n^2 attention.

▲ashkankiani 16 hours ago | parent | prev [-]

he said specifically lossless attention. As in that algorithm. what you just said is irrelevant to the conversation because all of the current AI hype is built on the back of attention

▲vlovich123 4 hours ago | parent [-]

Here’s what was literally quoted in the response:

> That compute will still be useful even if we find a more efficient architecture

Fixing N^2 attention would require a different architecture, but the person I was responding to was treating attention as if it’s the only architecture we have for intelligence when we know that not only do we have less competitive intelligence for cheaper, human intelligence doesn’t use n^2 attention. You have also fallen into the trap of misunderstanding the statement same as the person I replied to.