But they were! My college student ID from 1980 has my SS number right on it. I have here an employee badge from the Walt Disney Company from 2000 and my SS number is in a bar code right on the bottom. Even in 2000, it would be trivial to take a photo of someone wearing the ID badge and decode the 1D bar code on it.

Right, but somehow a bunch of companies still made them into a "prove who you are" factor, and (if we assume incompetence instead of malice) that means a bunch of decision makers still considered the a trust factor.

In some states, until the early 1990s, SSNs were also used as drivers license numbers.