prometheus1992 an hour ago

Does this mean they ended up sharing those private codebases with OAI, Anthropic etc? Also, the ~30% number tracks with my experience. I thought I was going insane for expecting too much from the models but they are still bad, including astra. This morning it messed something pretty trivial while fixing an issue which I was shocked to see. Also2, benchmarks don't mean much these days.

▲ ramigb 29 minutes ago | parent [-]

Can you please share, if you are comfortable of course, what did the model(s) mess up? what were you using codex/cc/pi? did the project have a solid agent.md/claude.md? I am genuinely curious whenever someone have such a low success rate with models what is happening because it could be fixed maybe? From my own experience using agents for the past year or so. The rate if I have to guess, is well above 70%. I mainly use claude (opus) on typescript react projects that are well setup with minimal plugins/MCPs!

happy to share more if you are interested.

	▲	irthomasthomas 3 minutes ago \| parent \| next [-]
		> For instance the task naming in the task file starts with an optimistic 1, 2, 3, 5, 5a but then eventually gets to 8a, 8a1, and then ends up with 8b2c2b3 and “8b2c2b2b checkpoint1”. The code that it produced got ever more wild. I don’t want to bore you with what it tried to build, but here are some example pieces of the interpreter changes: `Hardcoded constants everywhere Multiple same-line macro invocations in C Random indexes in production code Hideous tokenizer code in C` https://lucumr.pocoo.org/2026/9/7/astra-why/
	▲	dennisy 3 minutes ago \| parent \| prev [-]
		The reasons are highly project specific. The closer your project is to CRUD, the higher the chance of success.