I've been finding that too with these models

For example pointing them at a crash dump they manage to piece together and extract information I would not even try to extract by just looking at the raw memory

It's the same thing is decompilation and hacking. It's not that a human couldn't find the same bug if they looked, but a human wouldn't even try because of the low probability of success.

I once sent Gemini Pro a hex dump of a dmp file and it told me what the application was doing when it crashed. It seemed surprising, but not that surprising given what I've seen already.