That’s why you’re supposed to submit an unused swab with the samples, so that they can make sure the swab itself isn’t the source.

That only works if both swabs suffered the same contamination. If the contamination is sporadic, then it won't show.