Isn't it easy to implement a threshold so it's only effective on holdings above a certain value, or exemptions for primary abode or something along these lines?

It's not difficult conceptually to come up with schemes to tax wealth which doesn't unduly harm non-wealthy individuals (however you would define that level for these purposes). It's just that for one reason or another these schemes are not implemented, and one of those reasons is (plausibly) the political influence of wealthy individuals