Average human threat perceptions simply aren't useful here. People will also make wild assumptions about what kind of catastrophic thing could happen in aviation and then happily enter their car to drive somewhere without a thought in the world. In fact noone thought about designing gasoline fuel tanks in a safe way before we had cars. Not even really until people started burning. If we're already thinking about transporting antimatter safely today, this kind of technology will probably have an even better track record than planes.