I don't consider investing $20k into hardware to run SOTA open models, that are far behind proprietary SOTA, to be competitive.

Even if open models were competitive, it's still typically going to be more expensive than a cloud provider because of low utilization and higher purchase price.

▲

shric an hour ago | parent | next [-]

Last time I checked it costs closer to 500K to run SOTA open models at any usable speed.

▲

kaladin-jasnah 2 hours ago | parent | prev [-]

How do I know that a private LLM system won't have a degradation of quality similar to this or worse? The only thing I can think of for the proposed scenario is some sort of homomorphic encryption system? But not sure.

	▲	r2_pilot 2 hours ago \| parent [-]
		>How do I know that a private LLM system won't have a degradation of quality similar to this or worse? You use benchmarks, you test, and because YOU'RE the sysadmin you know what weights are running at what time, it's very visible. You can airgap the hardware and be guaranteed it won't change over time. And, frankly, degradation over time doesn't seem to be what's happening with the open models.