I've been saying for a while that given a proper harness, small local models can perform incredibly well. When you have a system that can try everything, it will eventually get it right as long as you can prevent it from getting it wrong in the meantime.

▲

zambelli 5 hours ago | parent | next [-]

Lol, I love that framing. Yeah, the small models have impressed me a lot during this work. The reasoning can be quite good, and definitely sufficient for a lot of cases. Just gotta nudge em back on track Every now and then and they'll figure it out.

▲

cornholio 5 hours ago | parent | prev | next [-]

If I understood correctly, the model will get it right because it knows when it isn't right.

▲

zambelli 5 hours ago | parent | next [-]

Essentially, yes that's right! There's some subtlety in how to let it know it was wrong (returning things as tool errors because it trained on that), but that's the gist of it - sort of a self-correcting architecture.

▲

tomjakubowski 3 hours ago | parent | prev [-]

https://en.wikipedia.org/wiki/Apophatic_theology

	▲	jon_richards an hour ago \| parent [-]
		I was expecting this https://knowyourmeme.com/memes/the-missile-knows-where-it-is

▲

2 hours ago | parent | prev | next [-]

[deleted]

▲

koolba 4 hours ago | parent | prev [-]

A thousand monkeys on a thousand typewriters…

▲

zambelli 4 hours ago | parent | next [-]

That is the whole challenge, actually! A new metric I'm going to dogfood into forge is ETTWS - estimated time to working solution.

A simple retry loop around your whole workflow could, in some cases, be all you need. But it could mean many blind attempts to get through a workflow successfully. And hopefully there isn't a payment step partway through!

The fewer hard errors nix the whole workflow, the lower your ETTWS.

▲

DiogenesKynikos an hour ago | parent | prev | next [-]

This is a thousand unusually smart monkeys who speak every major human language fluently and are proficient in every major programming language, but sometimes still make bizarre mistakes and need to be put back on track.

	▲	jplusequalt 21 minutes ago \| parent [-]
		This is fun for you?

▲

4 hours ago | parent | prev [-]

[deleted]