i just asked luna to write a function that transformed a file path while upholding an invariant about lowercase file names. completed that task just fine. i see no reason it wouldn't work if i asked it for 99 other functions with that same constraint.

again, you're expecting to use it without steering. that's not how you get the most out of the smaller models. i don't want to say you're using them wrong, but you clearly are. it could be deficiencies in your harness as well which the stronger models mask. try pi.

▲

anonymous908213 a day ago | parent [-]

You are sorely missing the point. You made the constraint the specific and only focus of the task. The example from the OP pertains to non-filename-related-tasks that have an additional specification added on to them. If your filenames need to be lowercase for all tasks, not just specifically filename-related tasks, you start running into problems.

> you're expecting to use it without steering

If I have to manually "steer" every mistake it makes on every trivial task, instead of being able to deterministically eliminate classes of failures so that fixing the task one time ensures that failure never happens again on similar future tasks, it is in fact faster to just type the fucking code myself.

▲

slopinthebag a day ago | parent | next [-]

op's tests are underspecified one-shot tasks. it's entirely expected that smaller models perform poorly.

but your claim was about file name invariants when generating code, especially small amounts of code, which is not what they were doing with DeepSWE, so you've confused two different things.

anyways, this is a pointless argument. if you can't figure out how to effectively use smaller models in your workflow, fine. but don't come on here talking shit about people who can.

	▲	anonymous908213 a day ago \| parent [-]
		While OP's tests are indeed of the vague DeepSWE variety, I still reference them because the exact same failure mode occurs in my own tests even for trivial scope tasks, down to the level of having one single constraint for a variety of <10 LoC tasks. As you say, though, this is a pointless argument, and I'm not going to burn my existing benchmarks to try to convince anyone. Feel free to use inferior models and workflows if that's what suits you.

▲

za_creature 19 hours ago | parent | prev [-]

> it is in fact faster to just type the fucking code myself.

shhhhhhh, not so loud!