Safety factors account for uncertainty. Uncertainty the quality of materials, of workmanship, of unaccounted-for sources of error. Uncertainty in whether the maximum load in the spec will actually be followed.

Without a safety factor, that uncertainty means that, some of the time, some of your bridge will fall down