Remix.run Logo
▲ logicchains 18 hours ago

>That compute will still be useful even if we find a more efficient architecture

Mathematically there's no way to have lossless attention without O(n^2) cost; some magical massively more efficient architecture is not going to pop up the same way nobody's going to find a way to sort an arbitrary list in log(n) time. There may still be more opportunities for incremental gains (like using the bare-minimum number of full-quadratic-attention layers necessary), but they're not going to be orders of magnitude more efficient without sacrificing intelligence.