this is the closest benchmark to my experience using the model harness combo. Astra for as great as it is falls slightly behind Fable 5.1 for me for large feature work (although it comments code much better). in particular, Fable is able to assess priority better than Astra (meaning Astra sometimes does things that aren’t worthwhile while missing things that are clearly important, particularly on possible ballooning scenarios- fable catches “this works for x amount of data but if we run this on y way greater than x amount of data we’ll run into issues). Gemini 3.8 is under appreciated, use Google Stitch to see it in action if you haven’t used Agy yet.

▲

majormajor an hour ago | parent | next [-]

Do you find Fable significantly better than Opus at avoiding-overengineering? All of my recent testing of Anthropic models seems like they're tuned-to-hell to (a) be much slower than they need to be (running tests over and over during the loop vs at the end, say, even if those tests take a few minutes a pop) and (b) doing exactly that sort of "built a lot of fancy enterprisey feature-adjacent 'stuff'" even before nailing the actual feature. Sol and Terra both have some of the latter but they seem to do the actual work a fair bit faster (this may be a usage-based-priority-tier/rate-limit thing though) which helps offset it.

I think the bigco folks saw all the "it wrote all this code but the tests didn't pass" or "it wrote the feature but it's super brittle" and tuned the newer model+harness combinations incredibly aggressively to try to turn a lazy prompt into "median Enterprise Architecture design suggestions" to bring up the baseline, but in a way that slows you down if you don't want that.

I'm not on big enough subscriptions to want to burn a lot time just evaluating Fable/Astra comparatively until they're cheaper, heh. I can steer any of the cheaper ones just fine anyway.

	▲	visiondude 44 minutes ago \| parent [-]
		i think this likely depends on workflow. for me, the first step is always a plan file artifact on disc, which i heavily review and go back and forth until satisfied. i often have to split the plan into multiple phases because agents are still poor at assessing how large a change will be. sometimes before even starting the plan the task is to create a harness for validation (a way for agent to check their work). the codebase you drop them in also greatly changes how much “over” engineering they do. i think if you are good about reviewing plan files and managing scope for your agent these sort of issues fade away. in general though, gpt models are faster, more token efficient, better technical writers, and yeah i still find Fable 1.0 to be a step up even though it’s slower because it’s actually a tad smarter (and the more important and dynamic the change / feature the more fable 1.0 shows its slight edge over astra)

▲

retrobox an hour ago | parent | prev | next [-]

Anecdotally, +1. I’d also say this benchmark matches my experiences and how much I trust the model output

▲

ShellfishMeme 2 hours ago | parent | prev [-]

Astra constantly does this for me. It goes 90% of the way with some task but then skips the most important part. Then when told to please fix that and do it properly, it suddenly goes down a rabbit hole for 6h and fixes scenarios that aren't even relevant. It's awful at assessing what is important to do and what not, and where to ask for permission and where not.