Using agents is programming. Programming is done with the mind, the tapping of keys isn’t inherent to the process.

Unfortunately IDEs are not yet directly connected to our minds, so there's still that silly little step of encoding your ideas in a way that can be compiled into binary. Playing the broken telephone game with an LLM is not always the most efficient way of explaining things to a computer.