It changes the shipping address, not the billing address.

And yeah, I do tap it to change what card to use. "Every single time."