Very cool, thank you for putting it together and sharing the data

Would you mind adding Jeffy to your benchmark?

https://jeffyclassify.com/blog/when-llms-fail-at-classificat...

zlandes86 42 minutes ago | parent | next [-]

Okay, I tried it! It doesn't have a zero-shot mode, so the only way I could test it without training was with the SNLI head as judge. On all 5,240 prompts that stand-in ignored the number of people (pulling in about half the wordings whether 0 or 20 people were at stake) and contradicted itself 96% of the time. You could train a "trolley head" but that would just measure whoever labelled the training data!

▲

evanjrowley 4 hours ago | parent | prev | next [-]

Any relation to https://github.com/firelex/jeff ?

	▲	nico 3 hours ago \| parent [-]
		Hasn’t seen it, authors not related Looks very cool though. I’d definitely like to somehow “generalize” Jeffy to be more of a universal classifier. Although adhoc models pretty much always beat general ones at classification

▲

zlandes86 4 hours ago | parent | prev [-]

running it now!

	▲	nico 3 hours ago \| parent [-]
		Awesome! Super curious about how it fails, and then figuring out how to over fit, err I mean improve it, haha