new | show | ask | jobs Github

oscarfr 2 hours ago

Found this benchmark for Jev-class models: https://benchmarkheaven.com/jev-models

There are already many Jev-like models in there.

Edit: No affiliation. Just found it and thought others might find it interesting.

▲

jasonjmcghee an hour ago | parent | next [-]

The open source ones- I downloaded a number and tried them and compared to Jev.

Anything that required knowledge / familiarity mmBERT and ModernBERT post-trains performed much worse.

So it seems like they did some kind of useful expansive pre-training.

Things that were Qwen or Gemma Diffusion did better at those kinds of tasks but were generally pretty inconsistent in terms of whether they could succeed repeatedly (and be stable + reliable) on the many types of tasks that are in the cookbook part of the Jev docs.

If you ask Jev similar input + questions, it's pretty stable. And does a reasonable job on a lot of questions.

This one public benchmark (the only I've seen) seems to give the open versions way too much credit. It wasn't my experience at all.

It gave my a false wrong sense of what might be required to get it working for something at work to avoid needing a new subprocessor as - at least on Cloudflare / OpenRouter Jev is third-party not hosted.

▲

oscarfr 21 minutes ago | parent [-]

Thanks for sharing your findings!

We are working on running our own benchmark of Jev and some of the other models. Our use case is classification that runs in a UI. Currently LLMs have good accuracy, but are too slow (and expensive).

Jev not being available through a cloud provider (Bedrock or similar) makes it more challenging for us to start testing and rolling it out.

	▲	bglusman 10 minutes ago \| parent [-]
		I think I found Jev is now available through OpenRouter? Oh but maybe you mean somethign like Bedrock with zero-data-retention offered? Anyway, it was on OpenRouter the other day, lest anyone be confused by above, as I did some testing with it for my pareto frontier tool https://github.com/bglusman/model_skyline (which, apologies, is full of slop because its 100% AI maintained but, may or may not have some utility for guiding automated or manual decisions... it definitely showed that Jev was performing MUCH better than some of the open alternatives to it we tested anyway)

▲

raybb 19 minutes ago | parent | prev [-]

One thing I'm still trying to figure out is how this compares to something like gliner. If you're just doing classification in what situations would you choose kev vs gliner?