Historically speaking marriage did not mean a man agreed to be sexually exclusive with a particular woman.

It meant a woman agreed to be sexually exclusive with a particular man + various legal entitlements. The particular legal entitlements were what distinguished it from concubinage (which was also a woman agreeing to be sexually exclusive with a particular man).

Now that is not what it means in a modern context, but also in a modern context nearly every developed country has below replacement birth rates, meaning that these countries are dying so any particular claims of moral superiority or fitness of the marital/family structure norms of modern societies can basically be entirely disregarded.

Stop ignoring the part of the argument in which there was a promise made that was later broken. That is why it is immoral. Respond to that part.

So your excuse for cheating in marriage is declining birth rates? Absolutely hilarious.