Now, can you do it in <200ms for 45 questions at once, have 0% malformed output, and any kind of meaningful benchmark? We’ll wait!

Running on old home hardware, Jev is probably running on a very powerful cluster.

someguynamedq 6 hours ago | parent | prev | next [-]

Nothing has malformed output if you coerce it's output into a statically defined set of options

_davide_ 7 hours ago | parent | prev [-]

> <200ms for 45 questions at once

Considering your own question length: ~120 characters x 45 divided by 4.1 ~= 1317 tokens.

So question processing at 5.5k PP(around the actual PP speed of GPT5.6 Sol) it would take around ~0.24 seconds + the context processing.

Computing the output should be around ~20ms (at 50 tok/s), computing 45 tokens in parallel.

> have 0% malformed output

Pretty trivial; only the allowed output is selectable :)

So, I keep repeating myself: Jev was a low-hanging fruit all along; no one cared, and probably no one will in a few weeks?

haott 6 hours ago | parent | next [-]

Yeah but a lot of developers who didn't even know that this was a possibility now do, and will probably find use cases for it.

You can probably even share context between questions by cleverly manipulating the attention mask.

	▲	_davide_ 5 hours ago \| parent [-]
		Nice idea! Didn't think about that; a single linear memory allocation could do the trick