On the contrary, you can feed the model counterfactuals and see what it does. Edit the person to be white then feed the image back into the model and see if its opinion changes.

Maybe. I'm not so sure. If you change skin color, it might identify race by other aspects. If you stage the same situation with an entirely different person or otherwise digitally edit it, it might detect in some way that it's not a real situation or an altered image and respond based on that. They're black boxes.

I remember someone talking about AI detecting lung cancer from MRIs, and they discovered that it was assigning higher probabilities based on the age of the machine that took it (or maybe fMRI, and maybe it wasn't lung cancer, I'm forgetting the details). So those communities were poorer and had worse health and a higher incidence, or they'd have had the newer machine. You never really know how it's deciding anything (though in that case they did figure it out, it might not be impossible, but in the end, you probably end up figuring it out at a statistical level--I suspect the counter-factuals wouldn't be reliable), so you're introducing variables attempting to make a counter-factual, which makes me think that in the end, it'll come down to statistical analysis from real data/people/situations. Which is after you've let it loose doing the thing, unless you run it in parallel for a while before relying on it at all and statistically analyze that before relying on it. I'm not sure you can isolate a variable sufficiently except by statistical means on real data, because it looks at everything, and it might find a correlate to your correlate, and a correlate to that correlate, and reach the same conclusion based on your counter-factual not being able to sufficiently hide what it was looking at

I don't know, random thoughts