| ▲ | HeavenFox an hour ago | ||||||||||||||||
If they are literally sending the entire transcript and asking for a summary, wouldn't almost all of the tokens be cache hit, and therefore fairly cheap? | |||||||||||||||||
| ▲ | minraws an hour ago | parent | next [-] | ||||||||||||||||
I don't want to sound rude but even at cache hit you are burning almost input * 0.10 * context amount of tokens multiple times. If it happens 10 times you have already accepted the same cost as the entire session fresh prefill. People on HN are either too rich or wayyy too rich | |||||||||||||||||
| |||||||||||||||||
| ▲ | buzer an hour ago | parent | prev | next [-] | ||||||||||||||||
Didn't they change the cache TTL to 5 minutes earlier this year? So depending on exact timing it could potentially be invalidated between the updates. | |||||||||||||||||
| |||||||||||||||||
| ▲ | dietr1ch an hour ago | parent | prev [-] | ||||||||||||||||
Unless they prepend something like, "summarise this, " | |||||||||||||||||
| |||||||||||||||||