I was going to say that it's pretty much exactly like working with an experienced software engineer ... except it never seems to take a step back and look at the bigger picture. It will happily fix the same class of bugs over and over without realizing a bigger change is required.

That’s a direct result of how they’re fine tuned. RLHF and other mechanisms reward quick, locally correct answers that solve the user’s immediate problem. A reward for a solution like "take a two-day pause, rip out half the modules, and rewrite the core" just straight-up doesn't exist in training datasets

Wait, are you talking about llms or people?