Hosted models are big, and there is a lot going on behind the scenes that we users have no visibility into. OpenAI, Anthropic, Google, etc do much more than just feed raw prompt tokens straight into a big 1-2TB static model and pipe the output tokens back to the web browser. The result of this is that they can do more, and end-users can get away with a lot more in terms of vague prompts and missing background.

The biggest lesson I've learned working with local models so far is: with the smaller models, you have to understand their limitations, be willing to run experiments, and fine-tune the heck out of everything. There are endless choices to be made: which model to use, which quant, thinking or not, sampling parameters, llama.cpp vs vLLM, etc. They much more fiddly for serious work than just downloading Claude Code and having it one-shot your application. But some of us enjoy fiddling so it all works out in the end.

I've done zero fine tuning in the local models I use. I also didn't do a lot of experiments except asking the 4 or 5 I downloaded what version of x package was the newest. For my work flows small models are king.