Ah but as soon as you start serving clients there's maintaining a public API, juggling prompt caches from different people, etc. It's a completely different use case which vastly complicates your setup than just having one endpoint that's always there that does exactly what you need it to do with zero waiting or internet latency. It's not that complicated to set it up, a GPU or two, vLLM or llama.cpp and you're set, local APIs are drop in replacements.

Of course there's a better middle ground with renting a GPU VPS off Runpod or Vast or something, and you get most of the benefits already without having to buy currently overpriced hardware.