I don’t think it bothers me, as long as the audio isn’t stored… but I can definitely see why it would bother some people.

The device microphones are already always listening, this just makes use of that sound a lot more than just listening for “hey siri”.