Android has a button in the quick settings bar to enable/disable the microphone, which helps with this (as long as you trust the OS itself). I keep it disabled most of the time.