By stating that people's private business shouldn't concern others, you are also imposing a moral system on others. Throughout most of the history, and to many people even now, morality extends beyond what's observable to outsiders. See e.g. what most of the religions have to say on issues such as homosexuality or eating certain foods.

I'm not saying I have the right answer to all of this either, I'm just pointing out that your "morally neutral" stance isn't as neutral as you'd like to think.