With generated code, the directory structure, interface design and general state management is usually a haphazard mess. Even with the best frontier models. But what really gets me is the model often tries to make assumptions for me that I didn't specify in the prompt. Subtle things like which error states are "oh shit we need to bail" vs "this isn't a deal breaker." Sometimes it will ask, but more often than not it will just make a decision and it's often the wrong one. If I don't have a fully kitted out test suit and a good type checker to verify the final product against, the the whole looping thing is just useless to me and I'm back to reviewing every line of code it puts out and having to draw on my years of architecture experience to make sure we don't build a giant pile of trash.

> It helps to know that LLMs don’t “reason”. They predict ..

Semantics. Prediction is the training objective. The ability to reason can be, and very arguably is, an emergent property of that.

"They’re foundationally incapable of always and consistently preventing prompt injection attacks. “Alignment work”, safety harnesses, and sandboxes all help to add barriers against the worst, but there are fundamental gap" ...

They seem to be very good at a lot of rudimentary best practices, more so than humans, but more accurately - if you run and audit pass with specific instructions ... they're very good at that.

I mean - it's what they're the best at which is applying 'fuzzy heuristics' in a mechanical way. If can describe issues concisely, the patterns, the styles, the rules then LLMs can very mechanistically and methodologically grind through them.

I don't even see how this is controversial - without getting into 'what their reasoning means' - we can all agree that their synthetic reasoning is pretty good at narrow scales, and they've been 'trained by compilers' and are extremely good at spotting common patterns.

If you back that up with a lot of tokens ... they excel.

Designing architecture, that's difficult, but hammering away at all the 'known-knows across a system' especially to identify things ... they're pretty good at that.

> In the past year, agent harnesses crossed the “can it be done” rubicon.

Brother, I'm still in "Can you get it right?"-mode. What am I doing wrong? (Rhetorical, but advice welcomed).