This implies you didn’t run any sort of evaluations? It is not realistic for any sort of production use case to do this.

Agree. Switching models with a keypress is for developer coding.

Jev and this decisions api are mostly useful for inference at scale in a workload where cost and latency matter… and that’s where evals become crucial. Could coding tools use it? Sure, but that’s probably a special case.