That's highly unlikely that this is done flawlessly. This morning I found a bug in our error handling, I asked Sol to fix it. He rewrote the way the specific component that failed called the exception. I asked it to retry and go deeper, inside the library. It solution was to ignore the part of the message that caused the issue. Which worked very well, but wasn't the root cause (it was a serialization issue. It's always a serialization issue). It couldn't go deeper, always stuck trying to catch the error rather than fixing it for real. We found the root cause, fixed it, and it was done, one line change and 15 lines of test, in one hours. Instead of four hours of nothing except spending tokens.

But truly I'm mad at the vscode harness. I used to be able to follow the 'thought' of the AI model and steer them when I saw a mistake, it's now way harder to do as multiple research tasks are done in parallel.

Somehow I've never had an experience like that on Sol. I find it to never really "cheat" like that. I've had all my best results from Codex CLI though, maybe that's somehow different. I also don't go below xhigh for anything serious and if it's having an issue I escalate the thinking mode.