You're making a fair point. Any native pathogens would have been shipped back to Europe with slave populations.

The fact that Europe didn't have the same catastrophic population decline suggests that either that didn't happen (possible, but a stretch) or that Europeans already had immunity.

Which would only be true if there was some freak genetic immunity (also a stretch) or the disease was already in wide circulation (far more likely).