It's fast.

If you're comparing with something, you need to state 'fast' in relative terms. Jev is definitely fast, and if this Python takes the same time to get a decision then it's also fast. If it's 100* slower than Jev though, you shouldn't be calling it 'fast', because relatively speaking it's really, really slow.

▲

_davide_ 8 hours ago | parent [-]

By design it can't be significantly slower than Jev: the prompt processing (AKA PP) is exactly the same on both and will take most of the time. Then you can process every single "question" in parallel, just predicting one or two tokens (if an answer is ambiguous with a single token) per each question, again in a single batch.

So, fast in the LLM space and comparable with Jev.

	▲	ActivePattern 4 hours ago \| parent [-]
		That's right. There's only so much optimization that you can make to a transformer-based model and any tricks that Jev is employing, any open-source LLM can also employ.