I am positive I know exactly how this works, I made something similar a few months back. But the problem is without generation you are extremely limited in the use cases. And while the model can't hallucinate, it can still be wrong. It just can't make up data.

▲

dennisy 5 hours ago | parent | next [-]

Are you able to share how it works in that case?

▲

adroitboss 4 hours ago | parent [-]

I'll tell you this. Output isn't too cheap to meter, there is no decoder.

	▲	krackers 4 hours ago \| parent [-]
		So an encoder-only model with a classifier trained on the heads or something? DeepSeek recently switched to an encoder-decoder architecture in an attempt to get the best of both worlds (fast prefill while preserving generation capability), I wonder if that might be the future?

▲

mokre 5 hours ago | parent | prev [-]

That was the first thing that come into my head. OK I can train very simple model, that can generate json's for specific tasks, so what? How we can be sure that this "limited use cases" not just overfitting for particular outputs (or even distillation?)

Except this, this thing looks like revolution.