Another factor is if they're all from the same (sub)discipline. It doesn't take a lot of established researchers moving to shift a field's center of gravity somewhere else. When you start seeing research conferences that used to be in the US be held elsewhere instead, you'll know that the change has happened.

I'm on a researcher mailing-list discussing exactly this at the moment.

For the moment, the main argument for keeping some conferences within the US is the number of researchers (typically PhDs and postdocs) who couldn't attend then re-enter the US.

We'll see how that goes.