You're right - but in that instance the justification in my mind (and what you'd presumably argue if challenged legally) is that they're difficult to work with - which I'm okay with.

But not wanting to concrete that person's driveway because they're in the army, or because they're a politician or whatever else along those lines I think shouldn't be allowed.

Okay, I will now say "they are difficult to work with" or "i don't have time" every time a client walks up that I don't want to serve. Infact, that's exactly what a lot of handymen say already to avoid the irritation associated with declining a job. I hope you see how your "ought" will be difficult to implement in practice