That's a narrow take. Non-deterministic doesn't mean random; workflows can be reasonably validated and consistent to some known degree.

I work for an education department that serves a chatbot for students, and model changes go through painstaking content safety reviews. I initially assumed it's just a bunch of bureaucratic paranoia. But every other model upgrade has a measurably different adherence to the existing system prompts about not talking to the kids about sex and drugs and mental health issues.

The user cares about the distribution of outputs. That distribution is structurally determined by the distribution of inputs (i.e. prompts), the weights of the model, and (these days) the dynamics of the harness guiding successive generations.

The only way to characterize whether a choice is 'right' is to characterize the output distribution (i.e. evals)! Changing the underlying weights necessarily invalidates whatever characterization may have been done. One may assert that one's harness regularizes outputs back toward the desirable distribution, or one may hope the different weights induce a sufficiently similar output distribution.

But no, one should not be completely agnostic to the choice of weights just because there's some nondeterminism.

They’re nondeterministic at a fine level, but can be “deterministic” at a more general level: e.g. you might know that one model will always return properly formatted json when asked. That might not be true of the replacement, even if it is in general “better” and cheaper.

Just the risk of such a thing means regression testing every time you update the model, and you want to be able to run that testing on your schedule rather than having it forced on you.

    > those things are not deterministic

    > we implement end-to-end, bitwise batch-invariant, and deterministic kernels with minimal performance overhead

Crossing the street and Russian roulette both have non-deterministic risks of injury. And yet I would be bothered to find out that that on my way to work, I was playing Russian roulette by surprise.

A dice roll is non-deterministic.

Replacing a six-sided die for an eight-sided die also keeps rolls non-deterministic.

That doesn't mean it's fine to just replace the dice mid-game.

Nope, strong disagree. The model is one small part of the process harness; behaviors are usually routable with expected propensities. Unexpected model changes avoiding change management messes with monitoring and observability thresholds. Stochastic controls are a real thing when you have your distributions defined; your workflows on a new model will throw that expected prior out the window.

Models are not deterministic, but they do have a flavor. When that flavor changes it can change the nature of output in a way that is undesirable.

Sort of like shooting a rifle - where the bullets hit is (to some order of magnitude, no philosophizing please) non-deterministic, but different very similar rifles will group differently and need to be appropriately adjusted to hit anything.

It'd be very expensive to get a setup that can run non-flash well.

If they were still at original price I'd get a couple of DGX Sparks myself to run Flash models at a decent quant/context combo.

4.1 releases tomorrow, right now you're supposed to be served by same old model

Yes exactly. Automatic model downgrade seems horrible for a lot of production workloads, even if you are deterministically constraining the behavior of your agents.

relying on cloud models for anything that gives you ROI is tying a loose noose.

Strict determinism is a different, but related, issue.

E.g. if I've written a role playing character using a specific model I may want to pin the character to that model until I've been able to test the model being "better" doesn't affect the feel of the character before switching. That doesn't mean I need the character's responses to be completely deterministic, but that doesn't imply I'm fine with the character having a different quality or feel of response just because the new model is out.

It'd be nice if there was a more explicit way to signal in the request "I want what you think is best per dollar for this class of answer" vs "I want this model to answer".

It's very easy to tell who is not running production applications using these models based on comments like this

Yes you can and you should. Providers have SLAs for when models roll off support and this has been the case for APIs long before LLMs. For example https://platform.claude.com/docs/en/about-claude/model-depre... and https://developers.openai.com/api/docs/deprecations