Counterargument: this works for quick prototyping, but for any serious business, you will eventually develop a benchmark/eval to track how well the general model is working, and once you have that dataset, you might as well train a specific model

▲

woah 5 hours ago | parent | next [-]

Jev's bet is that if it works well enough for random use cases that nobody complains, then management won't feel a need to develop a benchmark/eval, and they won't need to employ all those data science guys.

▲

momojo 4 hours ago | parent [-]

I'd also add that they're hoping Jevon's Paradox also leads to a whole new segment of users who would have never reached for a classifier in the first place, given the barrier to entry.

	▲	sanderjd an hour ago \| parent \| next [-]
		Yes this is what I'm interested in. I think they might be right. I'm already finding myself thinking "well maybe a classifier would be useful here now that it's so easy to do...". This probably just means that I could have been reaching for that tool more often already. But in practice I wasn't, and this has opened my eyes to the potential opportunities there.
	▲	woah 4 hours ago \| parent \| prev [-]
		And if you do get complaints or feedback on the classification, have a dev log into the user's account, tweak the Jev prompt a little until the issue goes away, and push it to production

▲

ACCount39 5 hours ago | parent | prev [-]

Or not. And replace the generalist with the next generalist that gets you +15% on that benchmark for the same price, or gives you the same benchmark performance for half the price.

One advantage of using generalist models is that the generalists are improving - regardless of whether you're doing anything about it.