| ▲ | serf 5 hours ago | |
>.. nor any sort of agentic work. may I ask why? oai luna runs at 60-70 toks/s; astra runs around 35-45 , are you just worried that your agentic work can't spread across a lot of parallel work? | ||
| ▲ | kolinko 5 hours ago | parent | next [-] | |
The numbers you povided are for output/generation. Input is around 5-20 thousand for Astra, and 16-60k for Luna. So a few hundred times faster. Most of the tokens consumed by agents is reading stuff - docs, tool outputs, code etc. Previewing a content of a dumb single page of html is a few thousand tokens. If you use claude or some other harness you can ask it how many cache writes/input tokens are there in avg session, and how much time it would take if preprocessing speed was just 60 tok/s. | ||
| ▲ | derefr 5 hours ago | parent | prev [-] | |
I would guess the assumption is that "agentic work" should be happening asynchronously in the background on a machine that's also used for foreground interactive work, where the slower the inference is, the longer the machine's GPU and [V]RAM are "hogged" by it at a time, and so the more the machine's performance seems to degrade. | ||