This is an example of why I'm a little stumped about why people are excited about Jev-style decision models. For any application at scale it'd be way more accurate to have a dedicated classifier using the right architecture (neural net, bayesian, etc.) LLMs drive the cost of making those models down to near zero.

Just use the LLM to make a thing that's actually good at what you need it to do.

Jev's launch was the best AI era marketing campaign I've seen so far. It was executed and timed so perfectly that Jev became synonymous with decision models overnight.

I think it reveals how many people were only familiar with LLMs as the generic AI tool for everything. It only took one ambitious marketing campaign for someone to come along and present a simpler technology as a new breakthrough to audiences that hadn't considered anything else.

The largest LLMs are really amazing for how much they can accomplish on unseen tasks. That is undeniably remarkable. The mistake that gets made everywhere is thinking that they're the best tool for every task, because LLMs are the only AI tool in most developer's AI toolbelt right now.

There are a lot of people working in software that have limited knowledge of information theory or machine learning.

Unfortunately LLMs are magic to them.

When you lack a first principles understanding of how something really works, you will bolt on hacks and duct tape on the wrong approach.

> Just use the LLM to make a thing that's actually good at what you need it to do.

This is the end state of most imagined use cases for “agentic” software IMO. Use the AI to build software, with a classifier or LLM as the expansion joint in awkward spots.

People like me who are excited about Jev-style models aren't going to use them to diagnose cancer. We are going to use them to make fairly straightforward decisions very quickly (300 ms or less) and very cheaply ($0.042 per M input tokens - output tokens aren't billed). Even the lower end frontier models will take much longer and be much more expensive, so these decision models are perfect for certain things.

I do agree with you, but I do think you're also making the classic "curse of knowledge" mistake.

You have to know the right words to ask the LLM to make and train a classifier. You and I already know the words. But they are not actually that trivial to learn for an average joe, and people are lazy.

Prompted LLM classifiers are good for when training data is expensive and/or you don’t expect a high inference volume. It’s basically choosing lower model development costs in exchange for higher inference costs, which can be good for TCO, time to market, or both. Heck, I’ve worked on some applications that weren’t even feasible before LLM-prompted classification because of the cost to acquire enough data to train a supervised model.

Cost/benefit tips in favor of supervised machine learning when training data is cheap and/or plentiful, or when you expect to be able to amortize model development costs across a whole lot of inference.

This is like saying we don’t need Dropbox because we have FTP servers.

People vastly underrate the patience of people to learn and setup a classifier. Jev is one API call. Training a good classifier is hard.

I think there is a parallel between engineering systems (specialized) and commodity cloud hardware (general purpose).

While no doubt, you can get better performance using a specialized hardware … there becomes a breaking point when the commodity / general purpose is good enough while allowing you lots more flexibility as well as typically being cheaper that is become hard to justify the specialized solution.

I think the contention is CNN vs Transformer

There's a great bit of shortform video content floating around that shows a stammering younger guy in a welding mask:

  > Do you have two pieces of metal that need to become one? 
  > But if it became  two pieces again it wouldn't be a big safety concern?
  > If that's you call Bob's OK welding!

Classifiers can require a lot of examples to train correctly. You can take a decision model's outputs and apply platt scaling to arrive at a good decision boundary a lot faster.

Eventually, yes, it makes sense to move to a pure classifier model as it's probably a lot cheaper. But then you need to learn how to serve your own classifier model correctly. Compute costs are being driven up by demand to serve models and unless you're careful it's easy to use your GPUs less effectively than a company specializing in serving a model.

A classifier requires a ton of labeled data to be processed for it to be good. Jev and likes are zero shot classifiers where none of the data is required. Imo both are useful for different tasks. I am not going to use jev based models for compliance or any tasks where being wrong can bankrupt my business. For something trivial like sentiment analysis it’s a god sent.

Training a model requires labels training data. Getting up and running with an LLM just requires a prompt.

I agree that a good model will outperform an LLM, but the LLM is so much faster/easier to build. That’s the usual tradeoff.

> For any application at scale it'd be way more accurate to have a dedicated classifier using the right architecture

Agreed, and that’s the premise behind Jeffy, open source, bundles pretrained task-specific models that run (and train) locally on CPU: https://jeffyclassify.com/

At the same time, I’ve been working with some people implementing it, and there’s definitely value in having a more general model like Jev, or an LLM, for initial exploration and data labeling, to then use that data and train the task-specific model. It’s a whole workflow

Also, the market/pain insight of Typesafe is super interesting. They realized that a big chunk of the market was using LLMs as classifiers, and they built a solution around that (very much to your point, a task-specific model, rather than a more general LLM), and it worked! (Also they have the ability to generate a lot of attention, given the founder being ex-OpenAI and having raised quite a bit of money too), so it’s not just a technical/engineering thing, it’s a business/market/need/discoverability issue as well

Use LLMs to implement, process data, test and iterate statistical and NN models.

What we've been trying Jev for is interpreting things like short product descriptions provided by customers, and determine which of the natural language categories fits best.

The list of categories are provided by the government and can change several times a year, and the product descriptions are almost random (think the various titles of AliExpress listings of essentially the same product).

What kind of dedicated classifier would be better? Genuinely curious.

There's a significant engineering cost to having a custom classifier.

An off-the-shelf API has way less friction, if the results are good enough why bother?

>LLMs drive the cost of making those models down to near zero.

The last place I worked and my current job have a tremendously bureaucratic process for standing up new internally trained models. You could easily spend ten times as much mental energy and effort doing the paperwork and revisiting it annually as you do in the care and feeding of the model itself.

If a canned product meets the requirement, there's a lot to be said for using it. Even if it costs a bit more per decision, the all-in costs are WAY lower.

Once you have a hammer everything is a nail.

> Just use the LLM to make a thing that's actually good at what you need it to do.

I don’t have a deep understanding of this topic, but wouldn’t you also need a good amount of relevant data to train on?

> For any application at scale it'd be way more accurate to have a dedicated classifier using the right architecture

There’s lots of applications where it’s not being deployed at scale. Also lots of cases where someone who knows they need something classified does not know what a classifier is.

> why people are excited about Jev-style decision models.

I think its a combination of being new to classification models and not using in anger.

LLM Decision models are still relatively new and suffer from a number of issues. Order of choices impacts probability, it starts to break down over 20 classifications and even how you word things impacts probability.

Don’t those decision models mainly route to other models? My understanding is that they wouldn’t handle a request but forward it along with some kind of semantic capability.

I have been incredibly impressed by LLMs for coding and data analysis and was expecting the LLMs to perform better so I was surprised by the results especially since this is an old model and nowhere near state of the art for skin cancer diagnosis.

For some types of applications, Jev is also useful as a stepping stone, you use it for the cases it's most confident so you already get gains from using it, and then collect all the usage to train your specialized model.

They make very good classifiers and are easy to fine tune for specific use cases.

it's precisely because you can tune them to do that off a strong base

and because they start off generalized, you can train them against multiple of your workloads (they'll start with some generalized performance/world knowledge about them)

Which allows you to embed proprietary processes/data (world knowledge only you may have) across multiple domains into a model cheaply

> For any application at scale it'd be way more accurate to have a dedicated classifier

"at scale" or the above use case of "specialized domain"

Jev being cheaper than an a full LLM call creates places where other tools might work better later - when the volume or the available data warrants it.

System optimal, code optimal and cost optimal aren't always aligned. Pick the right tool for the job - and dont get married to it. On occasion worse is better, and moreso when your testing or it is a one off/disposable.

Internet discovers If statements

subnanosecond response times

model fits on CPU

The only real usecase I can see for jev style like models are processing long lists of items. i.e server logs or financial transactions or sensor data in robotics.

I mean, it's the pareto principle, right? 80% of the applications require 20% of the intelligence. If I can drop in something without needing to create high quality training data that's good enough, that's great! Skin cancer probably isn't one of those applications.

[dead]