the problem in my opinion comes down to a sort of poor "theory of mind" on the part of agents at our current moment.

The AI is really bad at guessing what its reader already will know. Not just before reading, but also from reading the text. This means that it so often explains things that don't need explaining, and emphasizes things that don't need emphasis.

It's not Y but X. except no sane person would ever think it was Y, and it was clear to everyone it is X already (or X is not a widely understandable thing in the first place, or the AI hasn't established a satisfying reason for why this distinction isn't important, or or or).

This is, I think, due to a combination of training on human feedback rewarding surface level good text (people marking preference on their 300th answer of the day are not trying to understand whatever point the agent is making they're just seeing whether it seems like it knows what its talking about) combined with later training by automated grading systems (other llms) that always have perfect context - if I also went and read all the code the agent referenced, read all of its 12 paragraph answers in gave at every turn of this back and forth conversation, knew all the files on the hard drive and had read all the diffs it made, then yeah it would be fine to assume some of the things it does, but I don't do that when im talking with my boss, because he asked me to do it so he doesn't have to understand everything, and he expects me to surface important stuff and minimise other stuff in a sensible way