With purely quadratic attention, the cost of the token at 372K is 87% more than the token at 272K.

There are attention mechanisms that help to mitigate this, but you can clearly see in the chart that Tibo posted that the attention mechanisms they are using are still scaling quadratically.

This becomes an optimization problem for token t:

min f[cost, quality loss] = compaction cost[t] + compaction quality loss[t] + token cost[t]

▲

conradkay a day ago | parent [-]

It's not quadratic attention, you get that curve from the input tokens going up linearly, since the graph is measuring cumulative cost at each token count. Basically for y=5 it's 5+4+3+2+1, or f(x) = x(x+1)/2

https://pbs.twimg.com/media/HNFc4Dma8AA76FW.jpg?name=orig

	▲	salamo a day ago \| parent [-]
		On the one hand, compacting at 300k saves OpenAI 40%. That's great. On the other hand, $10 for 1M tokens still seems really high? It's not too hard to blow through that in an hour or two.