Apply the same logical test to freedom of speech, and you’ll get the same result.

You’re not missing anything about what’s likely to happen to you personally. What you’re missing is the manner in which rights shape your life and your society even when you don’t exercise them, and sometimes even when nobody is currently exercising them, and that significant harm can be built out of a vast number of smaller harms that aren’t individually that bad.