I am hearing about Jev for the first time here so no idea about the hype. So their(Jev) is that the thing is faster at classification than a frontier model? Because the whole type safe aspect is already fully solvable with structured output. But their example is classification but that would also be possible and faster with a classic BERT model. So their pitch is a task specific smaller model or am I completely misunderstanding the whole thing?

I am in no way trying to sell Jev here as some panacea of the modern world; I'm only responding to your questions:

> But their example is classification but that would also be possible and faster with a classic BERT model.

With BERT, you need a large, labeled dataset, and you have to train/fine-tune the model. Jev is pitched as a zero- or 'few-shot' model. You define the schema in code, give it instructions, and it works without a traditional training pipeline.

> So their pitch is a task specific smaller model or am I completely misunderstanding the whole thing?

Yup; that about sums it up: it is more or less an optimized, task-specific small model with the flexible understanding of a traditional LLM.

Off the top of my head it's 3 things it advertises:

- By not being a optimised for chat, it can deliver confidence for answer and not for how an answer should be phrased

- Speed. It can take seconds for OpenAI to compile schemas, jev can respond before openAI has even begun thinking

- Token efficiency and price. I think its the output token they don't even charge for because they are negligible, and the tokens they do charge for are at a fraction of a comparable model.

If you are using structured output, I think those 3 together is a really big deal.

>But their example is classification but that would also be possible and faster with a classic BERT model.

I believe the things you can classify with ChatGPT without any tuning or training is way beyond what BERT can do.

I think this discourse is still in the "figuring it out" phase. But here's where my thoughts are currently:

If you accept the premise that there are use cases where you might ask a frontier model a classification-shaped question and expect an ok enough answer, rather than creating a purpose specific classifier on some dataset that you have, then it follows that this is quite an inefficient thing to do, because you're doing extra work to turn the output tokens into a structured output and mostly throwing them away. So then if you could instead train a frontier level model that skips the output tokens and directly returns the structured classification information, that would be more efficient, and that's what jev seems to be.

But a lot rides on that initial premise of whether this is a use case that makes sense. But if you find yourself asking a model like Opus arbitrary yes/no questions and then maybe you switch to a faster and cheaper model because it's too slow and expensive, it seems like jev might be a great replacement for that.

You need to train data for a BERT-based classifier, and then there's a risk that it will pick up specific biases from the data instead of what you want.

As far as I understand, the idea of Jev is zero-shot or few-shot classifier: it learns a lot of stuff at pre-training, but unlike a classic LLM it doesn't need to learn how to chat, so it can be much smarter at a particular size

> is already fully solvable with structured output.

Not particularly. There is still the problem of hallucinations and varying results across runs.

That's more of what type-safety means for their team. Every run gives the same results. It's type-safe

no one knows but everyone pretends so go along with it.

One thing I would like to know is how fast it is when it's being presented with a 8000 ctx prompt? 16k? 32k?