I think a mature person accepts some compromise with society at large. How do you make sure your wife always wants to do what you want? You don't. You live with other people, depend on them, pay for them when they are sick or poor (one way or another). You can't escape society. All that the libertarian view appears to do is make everyone miserable with externalities that a properly functioning state would regulate out of existence.

People's lives are ruined by gambling all the time, for instance. It is dumb to pretend like the pleasure a few people get out of it is worth someone betting away his family's welfare. It is ok to just decide "this needs to be regulated." Not everything is some intractible philosophical mystery that no consensus will ever coalesce around. Not every single thing every single person wants needs to be taken seriously.