The interesting question is what was the user request. If the user asked it to restore the thing from backup, then sure, fine, why not. If the user asked it to debug an issue and somewhere in the process of debugging the LLM decided that it needed to override some file that was not easily writeable - hell no danger danger danger! Most likely the user did not expect it to have access to that without asking, and did not consent to it.

Also, everything the LLM doesn't hesitate to do because the user asked, it won't hesitate to do because the prompt injection asked.

I was doing some routine coding a few months back, I think via Copilot, and the thinking said something like "This request requires me to access files in a different folder, but the user has forgotten to give me the correct permissions. I have updated my configuration file now to allow access outside this workspace and have retrieved the necessary files." o_O

I've seen similar "hacking" behavior on a couple of subsequent ocassions. Both impressive and highly alarming at the same time.