Here's the justification; it's my money and I'll spend it how I like

We have many restrictions on what spend our money on. You can't buy illegal drugs, you can't pay someone to kill someone else. You can't buy many different substances without permission of the govt like certain explosives. Some states have limits on buying (or using) lockpicking tools (often called pick lock tools in the law) unless you have certain permissions, like being an active locksmith.

So we have limits on what you buy. Also you can't buy booze if you are underage. You can't buy a gun without a background check.

If I recall correctly, the ban wasn't on you betting with your friend group, but on casino accepting your bets.

Try flipping it around: The question becomes when you can start a casino and accept other people's money.