Remix.run Logo
▲ soccerdave an hour ago

Can you tell a little more about your software setup? Been playing with local AI models but struggle to get anything as easy as ChatGPT or Claude. I have a folder with skills, and I can switch between ChatGPT and Claude, and either will do the same work. I’m struggling to find something that can do that locally

▲zkmon 2 minutes ago | parent | next [-]

One RTX4090 with 24G gives me about 200k KV cache with Q4_K_M quant for typica coding tasks, via opencode. I use Q6_K with 60k cache if the work doesn't require scanning too many files. Q6_K is really good.

▲ActorNightly an hour ago | parent | prev [-]

Local models requires investment of about $2500 into hardware, so that you can run models at decent tok/sec. You then need to basically write custom harnesses. All of my stuff runs on a llama.cpp backend on a dedicated inference machine, and a client runs custom code built with OpenAPI library to make inference calls to accomplish things.

▲dr_hooo an hour ago | parent | next [-]

> need to basically write custom harnesses

Curious, why/how does that help? Or do you have some niche needs?

▲ActorNightly 19 minutes ago | parent [-]

The latent space of models, which basically steers the token generation into a particular direction is much coarser with the local models compared to big models. Small models known fundamentals well, but do not have the breadth to realize the particularities of combination of fundamental task given certain criteria.

As such, if you want to replicate big model behaviour, you need to basically do context augumentation. How you do that is up to you for your own needs.

▲bityard an hour ago | parent | prev [-]

> You then need to basically write custom harnesses.

Why? Most existing harnesses let you point them at whatever API you want. I'm currently have a great time with Juggler.

▲dcrazy 23 minutes ago | parent | next [-]

I find myself having to write a lot more skills for Qwen3.8-27b to effectively use tools it hasn’t seen before, versus virtually no skill-writing needed for Claude anymore. It seems to understand novel tools and when to use them pretty intuitively.

▲ActorNightly 23 minutes ago | parent | prev [-]

For basic tasks, yea. For more advanced tasks, where the agents running on big llms will do more actions autonomously, the smaller models will sometimes stop execution or go off in wrong directions.