Now imagine the token/s rate decline after context fill at 200,000+ context.

Fair, and we didn't measure it.

Decode was flat from 128 to 512 generated tokens (0.926 → 0.923 tok/s drafter-off), but that's a 6-token prompt plus the output — total context under a thousand.

The current configuration admits about 4.4k tokens of context at all, and at anything like 200k the killer wouldn't be decode, it would be prefill: today it reads each layer's experts once per 64-row pass,

so 200k tokens of prompt would be measured in days, not minutes, until the scheduling fix.

▲

NooneAtAll3 3 hours ago | parent [-]

what's the main limitation on context size?

4.4k seems... I just realized I have no sense of scale whatsoever

	▲	Argonautlabs 3 hours ago \| parent [-]
		Memory, not the model. The KV cache on this engine grows about 2.8 MiB per token of context, and the machine's 128 GB is already holding the 50.7 GiB resident trunk plus reserves the speculative-verify path needs (we found the hard way that squeezing those makes the verifier reject wide batches and decode falls to single-token steps). With the current reserves the engine admits ~4.4k tokens; that's a configuration ceiling you can raise by giving the cache more of the 128 GB and accepting less headroom elsewhere. K3 itself supports far longer contexts — but see the prefill caveat above: on this setup long prompts cost minutes per 512 tokens until the scheduling fix lands.