I'm doing the same. My reason is not the IDE, I just can't let AI agent software onto my machine. I have no trust at all in it and the companies who make this software. I neither trust them in terms of file integrity nor for keeping secrets secret, and I do have to keep secrets like API keys on my file system.

Am I right in assuming that the people who use AI agent software use them in confined environments like VMs with tight version control?

Then it makes sense but the setup is not worth the hassle for me.