Remix.run Logo
▲ Topfi 11 hours ago

Purely for the models running your code base, have you ever ran an assessment of what that fragility costs you? Tool call error frequency, unnecessary test failures that the model then addresses but wouldn’t in a more parsable code base, inefficiency in navigation, anything of that sort? A model landing changes doesn’t mean it does it well.

I have seen all those when running a fairly new eval inspired by something I read in the ts-rust repo, giving a simple GPT-6 Astra code base to Opus 5.5.

Implementation accuracy was higher and token usage lower when Opus 5.5 got a Fable 5.1 built code base of the same feature set and coverage compared to the fragile GPT one. Incidentally I struggled to parse the latter but not the former.

Yes, a modern model will happily work on the most unmaintainable slop (doesn’t even need to be LLM derived slop, we have all seen bad Human written code) and that is great if you have an old code base to maintain. But if one can avoid it, even if one uses only models and never looks at the code, being efficient has value. Heck, rewrites are doable and affordable now, I have restarted projects and the gains were quickly apparent, even if we just look at model usage efficiency gains.

▲user43928 11 hours ago | parent [-]

No, since it seems difficult to assess, it's not like I have a handcrafted version of the same project to compare against.

▲Topfi 11 hours ago | parent [-]

Try another model just looking through the harness history and commits for blockers and errors, often eye opening.