Remix.run Logo
▲ segmondy 3 days ago

duh, this is not news. (general, fast and cheap) before decision models, you could pick only 2.

LLM as judges - generalized, but too slow. If you had to make millions of classifications a day, this will be the wrong approach. you won't/shouldn't use LLM to classify spam/no spam. hot dog/or something.

traditional classifiers, very specific 1 trick pony, super fast and cheap once built. If you need to make tons and tons of classifications, this would be the approach. but if you wanted a classifier right now for a novel problem, you need an expert to curate data, train and deploy.

decision models/jev - are generic, you can throw them at most generic classification problems, and they are good enough. it's a fine balance between general, fast and cheap. you get all 3

▲lostmsu 3 days ago | parent | next [-]

Is Jev any faster, cheaper, or more precise than medium LLMs like Luna 6?

▲ShinTakuya 2 hours ago | parent | next [-]

Yes. Not as fast/cheap as Typesafe claims, but lots of benchmarks suggest around 3-4 times cheaper, and 7-14 times faster.

- https://www.ml6.eu/en/blog/jev-vs-gpt-6-luna-vs-bert-text-cl... - https://tessl.io/blog/jev-is-136x-faster-and-27x-cheaper-tha... - https://x.com/fazxes/status/2100300097695232164 (this last one is Luna 5.6 but that isn't too different from 6 besides accuracy and cost)

▲soltanov 37 minutes ago | parent [-]

[flagged]

▲IanCal 2 hours ago | parent | prev | next [-]

Luna 6 is 10c per million input tokens and charges 5x that for output. Not sure if it’s still true but it used to be the case that structured outputs took time to process and cache which is relevant if the structure changes. It doesn’t give a percentage you can use for thresholds, and I’d want to know if Jen treats the questions as independent (they aren’t with luna, order of questions will change the result).

Jev is 4.2c/m tokens in and free out.

▲pokeapallascat 2 days ago | parent | prev [-]

not really

▲nicce 2 hours ago | parent [-]

I don't think Luna is fast enough by any means

▲soltanov 38 minutes ago | parent | prev [-]

[flagged]