That temptation is akin to giving in to maladaptive (day)dreaming. The problem is the dreaming, the noticing, the getting annoyed, and the feeling drained - all of these are happening at one place - being done by one entity - your brain. So how to train it to not to do that? Well, again, that same brain has to do it. What I am saying here is - our brains are extremely good at getting "used to". Because it "feels good" to the brain - at least in that instant. So any work that's happening on its own, feels good for most people. More the time passed in that "got/getting used to" state unchecked, more difficult it is to try and bring it back to not getting used to. Or control it, in an attempt to strike a fine balance.