There was never a time when it wasn't deeply frowned upon either, because having an affair is not the norm.

Do you really think it was never the norm for rich men to have affairs? That's a very optimistic view of history, I wish I shared it.