I am trying to estimate if my reaction to seeing GPT-5.6 Sol last on that list is reasonable or or mostly emotional and find that I have no way of telling.

▲

WD-42 an hour ago | parent | next [-]

Why would you get emotional over a model? They got you that good?

▲

CompoundEyes an hour ago | parent | prev | next [-]

I do think it’s the wizard not the wand at this point given a decent model. These benchmarks don’t have the wizard.

Otherwise I wouldn’t see others in the exact same codebase struggle and underutilize agents while others thrive using the exact same ones.

	▲	howunfortunate an hour ago \| parent [-]
		In other words, we're still in the era of centaur chess.

▲

didgeoridoo an hour ago | parent | prev [-]

Sol failing mostly on “unverified assumptions” and rarely hitting “integration errors” seems about right to me. I think Sol is second only to Astra (and miles ahead of even Fable) in architecting & engineering the right implementation — but only if you are extremely specific and provide tight guidelines and guardrails. If you give it a one-liner… you’re going to have a bad (SHA-256-hash-verified) time.

	▲	guybedo 12 minutes ago \| parent \| next [-]
		SHA-256-hash-verified sealed package artifact with automatic reconciliation system p95<0.5ms
	▲	enraged_camel 17 minutes ago \| parent \| prev [-]
		>> I think Sol is second only to Astra (and miles ahead of even Fable) in architecting & engineering the right implementation — but only if you are extremely specific and provide tight guidelines and guardrails. To me, having to give extremely specific instructions and provide tight guidelines and guardrails defeats the purpose of agentic coding agents almost completely. At that point I might as well do the task myself. With Fable I can start with a general ask like "I'm trying to do X, can you investigate and tell me what the shape would look like" and have it poke around and think, ask me questions with single-choice or multiple-choice answers, then break the task into small chunks, each of which becomes a ticket. With Astra, it's like pulling teeth. It often does not understand what I'm trying to do, takes things literally, does not go above and beyond (i.e. infer intent), and stops way too short of the actual goal. I have to constantly prod it and it's frankly exhausting.