No, all they had to do was come up with a quality post-training recipe, production inference stack that wouldn't fall over, GTM, documentation, schemas, etc. etc.

(also most signs point to this being LLaDA 2.0-adjacent so throw in solving some substantial mid-training)

I think it's 100% a hot take to call what they built trivial. Or at least it used to be.

There was a time when that kind of stuff was something between sour grapes and cluelessness about the gap between an idea and an actual commercial product deployed at scale, but now that's just weirdly normalized.

In fact, if anything I'm the weirdo for repeatedly taking issue with the way people are trivializing it ¯\_(ツ)_/¯

▲

janalsncm a day ago | parent | next [-]

Someone else posted the jevbench site which compares jev to a bunch of other models. If you look only at the accuracy dimension:

https://benchmarkheaven.com/jev-models?w=100-0-0-0#jevc-weig...

Jev actually isn’t anywhere near the top. It even loses to open weight clones. This tells me that whatever their “calibration” dataset is, it doesn’t seem to be anything special.

▲

BoorishBears a day ago | parent | next [-]

... why didn't you link to the actual benchmark which does have Jev at the top?

https://benchmarkheaven.com/jev-models

You linked to some weird subtable that labeled: " Not the default — not the JevBench Score", that can only be reached after you see what I just linked... lmao are you really this hard up about things?

Also every single question (even in the hard set) is single dimensional?: https://github.com/fstandhartinger/jevbench/blob/main/datase...

Jeeze, this is getting sad. I guess after all the mass-psychoses where people thought pointless things are going to change the world, we were due for a mass-psychosis where something interesting just has to be pointless?

▲

janalsncm a day ago | parent [-]

Because I was specifically responding to your claim that Jev’s training recipe would give it better accuracy than others. It doesn’t have better accuracy than others. You could do as well or better by distilling qwen for example.

Jev is ranked higher than others on the overall benchmark due to speed and/or cost, not accuracy.

	▲	brokencode 21 hours ago \| parent [-]
		Well yeah, that’s the whole idea. If speed and cost don’t matter, you could use Astra. Obviously it’s the speed and cost that make it compelling. The tradeoff is accuracy. Enough to matter? Maybe, maybe not. It’s not like it’s way down the chart. It’s probably good enough for a lot of tasks.

▲

ombansod 19 hours ago | parent | prev [-]

[dead]

▲

hodgehog11 a day ago | parent | prev [-]

Architecturally, it is trivial. That's something the community would have consensus on, so not a hot take.

I see your point, but Jev doesn't exist in a vacuum. When one (like me) says "trivial", they mean it relative to other attempts and developments in the field, all of which require everything you've mentioned at minimum. Commercialising any product, and doing it well, is hard. But the R&D factor here is substantially more straightforward than almost any other product in its category, because there is no architectural breakthrough here.

▲

BoorishBears a day ago | parent [-]

"all of which require everything you've mentioned at minimum"

Sorry who else did everything I mentioned? I think the guy behind Laya tried after noticing Jev's traction... but the site's auth went down and has stayed down for a day now.

"substantially more straightforward than almost any other product in its category"

More straightforward than the spite projects based on constrained decoding? Or Laya with it's couple of days post-training ModernBERT?

I have no doubt other teams can build models like this and I've love for a frontier lab to give us an even smarter model with these ergonomics... but in the rush to show Jev what's up, we're mostly getting slop.

PS: I don't know anyone who's done anything of note who uses trivial like that. The commentariat do, and the "I could have done that" crowd do, but I don't pay much attention to them until they actually do the thing.

	▲	hodgehog11 5 hours ago \| parent [-]
		By category, I meant other language models in general. The point of others putting something up to beat Jev is to show that, to date, no one has bothered to produce something like Jev, because anyone with decent LLM experience can roll their own for purpose with little effort and have been doing so for years. And can beat it on any metric you choose. Let me put it this way. OpenAI and Anthropic have a slight moat over the Chinese labs because they have strong training data and the most advanced RL strategies. It will take the Chinese labs significant R&D effort to bridge that, especially in math (and there is a good chance they will, provided they want to). Jev has no moat other than the fact that no one else has bothered to package a model in this way. Another lab could build a strong competitor very quickly if they want to put the effort in. That's the point of this post. There is no uncertainty about what they have done, nothing to figure out. Someone just needs to do it. I'm not sure what to say if you can't see the difference between the two. Jev is worth celebrating because of the idea to package it in this way. But it is not a paradigm shift and that is likely a problem for them.