| ▲ | duggan 4 hours ago | |
It was a real facepalm moment when I realised we were busting the cache on every request by including date time near the top of the main prompt. Even just moving it to the bottom helped move a lot of our usage into cache. Probably went from something like 30-50% cached tokens to 50-70%. | ||