If your card is stolen you should, but not necessarily if you fall for a Facebook ad that ships you a pile of rocks or a paper photo of the product you thought you bought.

Isn't that exactly when you should have your card number changed? You gave your card info to a blatant fraudster. If they're willing to ship you a pile of rocks then there's a significant chance they're willing to use the card info you gave them to make fraudulent purchases.