This seems to be a general chat app, but otherwise small models can be very effective within the right use cases and orchestration.

▲

embedding-shape 6 hours ago | parent [-]

> otherwise small models can be very effective within the right use cases and orchestration

very limited amount of use cases, perhaps. As a generalized chat assistant? I'm not sure you'd be able to get anything of value out from them, but happy to be proven otherwise. I have all of those locally already, without fine-tuning, what use case could I try right now where any of those are "very effective"?

	▲	dgb23 3 hours ago \| parent [-]
		Judging from my experimentation with local models: You can use a small coding model to produce working code with a deterministic workflow (ex: state machine) if you carefully prune the context and filter down what it can do per iteration. Instead of letting it "reason" through an ever growing history, you give it distinct piecemeal steps with tailored context. I think this can be generalized to: Anything that can be built from small, well understood pieces and can be validated and fixed step by step. Then the challenge becomes designing these workflows and automating them. (I'm not there yet, but one thing I have in mind might be a hybrid approach where the planning is produced by a more expensive model. The output it has to produce are data driven state machines or behavior trees (so they can be validated deterministically). Then it offloads the grunt work to a small, local model. When it's done, the work gets checked etc.)