It is cool but local models while okay already feel noticeably worse than even the cheapest APIs so I can't see myself sacrificing even a little bit of their quality for speed. I'm sure it's worth it for some usecases, curious to hear specific ones that people are already planning to deploy to production.

▲

Mashimo 2 hours ago | parent [-]

Maybe writing / bootstraping unit tests?

Does not need opus level to write, and easy to iterate on.

	▲	SkitterKherpi 2 hours ago \| parent [-]
		I can see it but even if I do that for something like tests I'd still eat the time cost of the normal Gemma for 10% extra performance. And further, if you switch between the fast and normal Gemma for different tasks you eat the big time cost of loading the other model (and maintaining both in the first place).