| ▲ | cyanydeez a day ago | |
I've been fighting the urge to keep tweaking the setup, and with local models, the desire is even more involved. I'm pretty close now to the following: 1. llamacpp with custom reasoning budget and reasoning message tailored to a specific agent. 2. opencode with tailored agent that has access to dynamic context pruning and a graph of the code base. 3. Because local models love to reason instead of working, the llamacpp budget stops the reason with a message that reminds them to use subagents and to prune the context --- This is successful enough that I can make and expand greenfield projects into full blown tools. I've just built this morning a open document templating system that lets me build reports into reports and make them look nice with templates. What would have bloated up to ~1M context stays within the dynamic context prune's limits set 50k-85k; some of the limits are because of the 128GB GPU, but mostly because processing times go down the larger the context. So yeah, it's pretty easy to want to keep tweaking this stuff. I'm happy though, I don't have to ever contemplate some cloud service changing my work flow because they need to squeeze $$$ out of my job. | ||