The roadblock to this is you seem to have to build it yourself. I've noted that none of the current cloud models are very good at building a replacement for themselves, and there's significant work that needs to be done to make a local LLM reliable in any way. I haven't found a single standalone package that makes setting them up easy. Sure, I can run Hermes Agent and a model, but getting the self-reflection loop in and all of the other stuff the need to actually be good? I'm still at it, trying to get anything to work reliably and factually.

Could be an opportunity for a business? Except nobody ever wants to pay for software