I've noticed that too with Sol (xhigh) it starts with pretty good architectural goals, but really get lost in the weeds with certification and validation. Like it won't trust native tools like (in my case packer, Ansible, gcloud) to fail reliably under possible error conditions, and insists on implementing custom verification.

It also doesn't have a clear idea of what the actual threat model is, and builds all kinds of extremely defensive systems to account for imagined hostile actors. I'm like "Dude, it's only our systems that are creating these SVGs, they're never going to be user supplied, so you don't need to write an entire validation and sanitation framework here."

It also seems to treat the desired initial state of something as a permanent invariant and designs elaborate tests to ensure that it remains that way. Then when you make one little change it has to go and update a ton of tests it created.

I've had to rip out a bunch of overengineered jank from several feature implementations, and in doing so I ended up having to create retrospective documents that warn against this kind of behavior that I'll have the model review whenever a plan begins to go sideways.

“Extreme defensiveness” is a good characterization.

I wonder if it’s an artifact of OpenAI’s values or rl training approach.

Also, it prob does make it perform better just not more efficient.

Great for the OpenAI employee working on security scanning who doesn’t have to pay for their tokens.

Not so much for the dev building their web app who is trying maximize their subscription.

Like hiring an aerospace engineer to build you a shed.