"They’re foundationally incapable of always and consistently preventing prompt injection attacks. “Alignment work”, safety harnesses, and sandboxes all help to add barriers against the worst, but there are fundamental gap" ...

They seem to be very good at a lot of rudimentary best practices, more so than humans, but more accurately - if you run and audit pass with specific instructions ... they're very good at that.

I mean - it's what they're the best at which is applying 'fuzzy heuristics' in a mechanical way. If can describe issues concisely, the patterns, the styles, the rules then LLMs can very mechanistically and methodologically grind through them.

I don't even see how this is controversial - without getting into 'what their reasoning means' - we can all agree that their synthetic reasoning is pretty good at narrow scales, and they've been 'trained by compilers' and are extremely good at spotting common patterns.

If you back that up with a lot of tokens ... they excel.

Designing architecture, that's difficult, but hammering away at all the 'known-knows across a system' especially to identify things ... they're pretty good at that.