> So, there's an ojective measure of shade of gray?

Very much no. More that almost all actions have good and bad impacts and how good or bad those impacts are can be judged completely differently between large groups of people.

To someone that's Hindu, eating a cow is a supremely bad act. For your standard meat eater, it's just lunch. The cow eater is only a "bad person" to the Hindu. To them, the act is likely a white or light gray act.

Now apply this logic to human slavery. The "good impacts" are what profits the slave holders, what makes their lives easier at the expense of the enslaved. Apply it to child labor and labor exploitation in general before labor laws. Then you'll see one side is behaving in a greedy manner, ignoring the humanity of the other side.