I haven't evaluated this one, but for what I have, it seem the pplx-decider-v1.2 (and its -flash variant) are pretty good. Anyone evaluated all these open-source decision models on their own tasks? (I evaluated on MonitoringBench).

▲

GavinEke 4 hours ago | parent | next [-]

I didn't look specifically at open-source but I evaluated as many decision models as I could on openrouter against how good are they at detecting phishing, which isn't the best use of these models but I was interested to see how they compared. https://emdashcyber.com/phishnchips-decision-models/

▲

theanonymousone 3 hours ago | parent | prev [-]

I had my first use case recently, and did some very non-systematic benchmarking of models on OpenRouter.

The only one that came close to Jev itself was the one from respan (respan/span-02), which is also open source and cheaper too.

	▲	mgrandl 2 hours ago \| parent [-]
		Have you checked mammoth-1/elephant-1 models? They are benchmarking quite well compared and are currently free on openrouter. I currently use mammoth-1 for a small internal app and it works flawlessly for what I am doing. https://huggingface.co/baselabs/mammoth-1