| ▲ | voiceeh an hour ago | |
Not if your use case needs speed. For one of my products I can't use an LLM that has a p99 of >700ms for TTFT. | ||
| ▲ | hansvm 20 minutes ago | parent [-] | |
If it could output 1k tokens per second but needed 4 seconds to produce the first batch of 4k, would that not be viable? | ||